Autonomous driving vehicle preparation and retrofit monitoring for efficient fleet management

Abstract

A system for autonomous vehicle preparation and retrofit monitoring for efficient fleet management includes sensors equipped on a vehicle. Each sensor has a different field of view of the vehicle or the surroundings of the vehicle. A human-machine interface (HMI) accessible to a fleet manager, the cloud computing server, and the controller communicate with each other and with the vehicle. The controller executes a fleet management application (FMA) having a first control logic for authorized personnel detection (APD), a second control logic for cargo and retrofit (CU) management, and a third control logic for vehicle daily preparation (VDR) management. The outputs from the first control logic, the second control logic, and the third control logic are transmitted to the HMI and a cloud computing server, and the FMA suggests a vehicle stop time mitigation policy to a fleet manager based on the outputs from the first control logic, the second control logic, and the third control logic.

Description

Technical Field

[0001] This disclosure relates to systems and methods for monitoring vehicle status, and more specifically, to systems and methods for monitoring and reporting vehicle access, location, modification, and mission readiness status. Background Technology

[0002] With vehicles being used more frequently and extensively for the delivery of goods and services across various business sectors, fleet management is becoming increasingly important. Understanding vehicle status, location, and downtime is crucial for effective fleet management. While current systems and methods for monitoring fleet vehicle status have achieved their intended purpose, there is a need for new and improved systems and methods that utilize pre-existing or retrofittable hardware to reduce vehicle downtime, optimize vehicle lifespan, decrease the likelihood of unauthorized vehicle use, reduce the possibility of vehicle and vehicle content tampering, and track vehicle usage without increasing system complexity. Summary of the Invention

[0003] According to several aspects of this disclosure, a system for monitoring the preparation and modification of autonomous vehicles for efficient fleet management includes at least one vehicle. Multiple sensors are mounted on the at least one vehicle and collect real-time information about the at least one vehicle and its surrounding environment. Each of the multiple sensors has a different field of view of the at least one vehicle or its surrounding environment. The system also includes a human-machine interface (HMI) accessible to a fleet manager, a cloud computing server, and a controller having a processor, memory, and one or more input / output (I / O) ports. The I / O ports communicate with the multiple sensors, the HMI, and the cloud computing server. The processor executes program control logic stored in the memory. The program control logic includes a fleet management application (FMA). The FMA has: first control logic for authorized personnel detection (APD); second control logic for cargo and modification (CU) management; and third control logic for vehicle daily preparation (VDR) management. Outputs from the first, second, and third control logic are transmitted directly or indirectly to the HMI and the cloud computing server via the I / O ports. Based on the outputs from the first, second, and third control logics, FMA generates suggested vehicle downtime mitigation strategies for the fleet manager.

[0004] In another aspect of this disclosure, the first control logic also includes control logic that causes the fleet manager to populate a database of personnel who are authorized users of at least one vehicle. The database is hosted on a cloud computing server.

[0005] In another aspect of this disclosure, the first control logic further includes control logic for monitoring at least one vehicle and its surrounding environment using multiple sensors, and control logic for determining that at least one vehicle is parked based on data from the multiple sensors. The first control logic also includes control logic for detecting motion within one or more fields of view of the multiple sensors, and control logic for determining that someone is approaching or inside at least one vehicle.

[0006] In another aspect of this disclosure, the first control logic further includes control logic that analyzes sensor data and compares a person approaching or inside at least one vehicle with information on authorized users stored in a database of authorized users of at least one vehicle, and control logic that generates a notification to the HMI based on abnormal or unexpected behavior of authorized personnel interacting with at least one vehicle. The first control logic also includes control logic that proactively and immediately generates a notification to the HMI, notifying the fleet manager that an unauthorized person is interacting with at least one vehicle, continuously recording actions taken by the unauthorized person, compiling a detailed report of the unauthorized person's actions, and control logic that generates a suggested response for situations where authorized or unauthorized personnel use at least one vehicle without authorization.

[0007] In another aspect of this disclosure, the second control logic further includes control logic for determining that at least one vehicle is being modified based on data from multiple sensors. The second control logic also includes control logic for continuously monitoring at least one vehicle and its surrounding environment using multiple sensors, and control logic for analyzing data from the multiple sensors and comparing the data with predetermined modification target information.

[0008] In another aspect of this disclosure, the control logic for analyzing data from multiple sensors also includes one or more algorithms selected from artificial intelligence (AI) algorithms, machine learning (ML) algorithms, and deep neural network (DNN) algorithms, including object detection algorithms; and compares the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle with predetermined modification targets for each modification purpose selected by the fleet manager.

[0009] In another aspect of this disclosure, the second control logic further includes control logic for determining the modification completion status and generating a notification to the HMI to inform the fleet manager of the modification completion status. The second control logic also includes control logic for generating and transmitting a report to the HMI, the report including information about the completed modification work, including: adding, removing, altering, or moving materials, tools, supplies, equipment, and fixtures in or on at least one vehicle, and control logic for selectively reporting whether the modification work on at least one vehicle is completed or not, based on whether the modification objective has been achieved.

[0010] In another aspect of this disclosure, the second control logic also includes control logic that, when it is determined that the modification target has not yet been achieved, adopts a suggested vehicle downtime mitigation strategy to reduce the downtime or idle time of at least one vehicle that has not yet been fully modified from Level 1 to Level 2 (below Level 1) by: notifying the fleet manager that for at least one vehicle that has not yet completed modification work, there are one or more potential alternative use cases for each vehicle; and causing the fleet manager to select at least one of the one or more potential alternative use cases.

[0011] In another aspect of this disclosure, the third control logic further includes control logic for receiving commands from the HMI from a fleet manager, the commands assigning one or more tasks to at least one vehicle, and control logic for determining, based on data from multiple sensors, that at least one vehicle possesses the appropriate equipment to perform the assigned tasks. The third control logic also includes control logic for activating multiple sensors and enabling the sensors to monitor at least one vehicle and its surrounding environment, as well as control logic for analyzing data from the multiple sensors and comparing the data from the multiple sensors with predetermined requirements for the assigned tasks.

[0012] In another aspect of this disclosure, the control logic for analyzing data from multiple sensors also includes one or more algorithms selected from artificial intelligence (AI) algorithms, machine learning (ML) algorithms, and deep neural network (DNN) algorithms, including object detection algorithms; and compares the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle with predetermined requirements for the task of distribution.

[0013] In another aspect of this disclosure, the third control logic further includes control logic that determines the daily readiness status of a vehicle and generates a notification to the HMI based on the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle, and predetermined requirements for task allocation, notifying the fleet manager of the daily readiness status of at least one vehicle. The third control logic also includes control logic that selectively generates and transmits to the HMI a report, including a "Check Complete" notification or a "Ready" notification, when at least one vehicle meets the predetermined requirements for task allocation; and control logic that selectively generates and transmits to the HMI a detailed report on the task allocation readiness status of at least one vehicle, the report relating to the current status of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle, when at least one vehicle does not meet the predetermined requirements for task allocation.

[0014] In another aspect of this disclosure, a method for monitoring the preparation and modification of autonomous vehicles for efficient fleet management includes collecting real-time information about at least one vehicle and its surrounding environment using multiple sensors equipped with at least one vehicle, each of the multiple sensors having a different field of view of at least one vehicle or its surrounding environment. The method also includes a fleet manager accessing a human-machine interface (HMI) and executing program control logic stored in the memory of a controller having a processor and one or more input / output (I / O) ports communicating with multiple sensors, the HMI, and a cloud computing server. This program control logic includes a fleet management application (FMA). The FMA includes control logic for performing: authorized personnel detection (APD); cargo and modification (CU) management; and vehicle daily preparation (VDR) management. The FMA also includes control logic for generating and transmitting outputs from APD, CU management, and VDR management, directly or indirectly, to the HMI and the cloud computing server via the I / O ports, including suggested vehicle downtime mitigation strategies that can be viewed by the fleet manager.

[0015] In another aspect of this disclosure, the method also includes having a fleet manager populate a database of persons who are authorized users of at least one vehicle, the database being hosted on a cloud computing server; and using multiple sensors to monitor at least one vehicle and its surrounding environment. The method also includes determining that at least one vehicle is parked based on data from the multiple sensors; detecting movement within one or more fields of view of the multiple sensors; and determining that someone is approaching or inside the at least one vehicle.

[0016] In another aspect of this disclosure, the method also includes analyzing sensor data and comparing the person approaching or inside at least one vehicle with information on authorized users stored in a database of authorized users of at least one vehicle; and generating a notification to the HMI based on abnormal or unexpected behavior of authorized personnel interacting with at least one vehicle. The method also includes proactively and immediately generating a notification to the HMI notifying the fleet manager that an unauthorized person is interacting with at least one vehicle, continuously recording actions taken by the unauthorized person, and compiling a detailed report of the unauthorized person's actions; and generating a suggested response for unauthorized use of at least one vehicle by authorized or unauthorized personnel.

[0017] In another aspect of this disclosure, the method also includes determining that at least one vehicle is being modified based on data from multiple sensors; and continuously monitoring at least one vehicle and its surrounding environment using multiple sensors. The method further includes analyzing the data from the multiple sensors and comparing it with predetermined modification target information by executing one or more algorithms selected from artificial intelligence (AI), machine learning (ML), and deep neural network (DNN) algorithms, including object detection algorithms, to compare the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle with predetermined modification targets for each modification purpose selected by the fleet manager.

[0018] In another aspect of this disclosure, the method further includes determining the modification completion status and generating a notification to the HMI to inform the fleet manager of the modification completion status; and generating and transmitting a report to the HMI including information about the completed modification work, including: adding, removing, altering, or moving materials, tools, supplies, equipment, and fixtures in or on at least one vehicle. The method also includes selectively reporting whether the modification work on at least one vehicle is completed or not, based on whether the modification objective has been achieved.

[0019] In another aspect of this disclosure, the method also includes employing a suggested vehicle downtime mitigation strategy when it is determined that the modification target has not yet been achieved, to reduce the downtime or idle time of at least one vehicle that has not yet been fully modified from Level 1 to Level 2 (below Level 1) by: notifying the fleet manager that for at least one vehicle that has not yet completed modification work, there are one or more potential alternative use cases for each vehicle; and causing the fleet manager to select at least one of the one or more potential alternative use cases.

[0020] In another aspect of this disclosure, the method further includes receiving an order from a fleet manager via an HMI to assign one or more tasks to at least one vehicle; and determining, based on data from multiple sensors, that at least one vehicle is appropriately equipped to perform the assigned tasks. The method also includes activating multiple sensors and having the sensors monitor at least one vehicle and its surrounding environment; and comparing the data from the multiple sensors with predetermined requirements for the assigned tasks by executing one or more algorithms selected from artificial intelligence (AI), machine learning (ML), and deep neural network (DNN) algorithms, including object detection algorithms, to compare the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle with the predetermined requirements for the assigned tasks.

[0021] In another aspect of this disclosure, the method further includes determining the daily readiness status of a vehicle based on the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle, and predetermined requirements for task allocation, and generating a notification to the HMI to inform the fleet manager of the daily readiness status of at least one vehicle. The method also includes selectively generating and transmitting to the HMI a report, including a "check completed" notification or a "ready" notification, when at least one vehicle meets the predetermined requirements for task allocation; and selectively generating and transmitting to the HMI a detailed report on the task allocation readiness status of at least one vehicle, the report relating to the current status of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle, when at least one vehicle does not meet the predetermined requirements for task allocation.

[0022] In another aspect of this disclosure, a method for monitoring the preparation and modification of autonomous vehicles for efficient fleet management includes collecting real-time information about at least one vehicle and its surrounding environment using multiple sensors equipped with at least one vehicle, each of the multiple sensors having a different field of view of at least one vehicle or its surrounding environment. The method also includes a fleet manager accessing a human-machine interface (HMI) and executing program control logic stored in the memory of a controller having a processor and one or more input / output (I / O) ports communicating with the multiple sensors, the HMI, and a cloud computing server. The program control logic includes a fleet management application (FMA) that includes control logic for: performing authorized personnel detection (APD); performing cargo and modification (CU) management; and performing vehicle daily preparation (VDR) management. The APD includes control logic for enabling the fleet manager to populate a database of personnel who are authorized users of at least one vehicle, hosted on a cloud computing server; monitoring at least one vehicle and its surrounding environment using multiple sensors; and determining that at least one vehicle is parked based on data from the multiple sensors. The APD also includes using multiple sensors to detect movement within one or more fields of view of multiple sensors; determining that someone is approaching or inside the at least one vehicle; and analyzing sensor data and comparing the person approaching or inside the at least one vehicle with information on authorized users stored in a database of authorized users of the at least one vehicle. The APD also includes control logic for generating notifications to the HMI based on abnormal or unexpected behavior of authorized personnel interacting with the at least one vehicle; and proactively and immediately generating notifications to the HMI to inform fleet managers that unauthorized personnel are interacting with the at least one vehicle, continuously recording actions taken by unauthorized personnel, and compiling detailed reports on unauthorized personnel. The APD also includes generating suggested responses for unauthorized use of the at least one vehicle by authorized or unauthorized personnel. The CU management control logic also includes control logic for determining that the at least one vehicle is being modified based on data from multiple sensors; and continuously monitoring the at least one vehicle and its surrounding environment using multiple sensors. The CU management control logic also includes control logic for analyzing data from multiple sensors and comparing the data from multiple sensors with predetermined modification target information by executing one or more of artificial intelligence (AI) algorithms, machine learning (ML) algorithms, and deep neural network (DNN) algorithms, including object detection algorithms, to compare the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle with predetermined modification targets for each modification purpose selected by the fleet manager.The CU management control logic also includes control logic for determining the modification completion status and generating a notification to the HMI to inform the fleet manager of the modification completion status; and generating and transmitting a report to the HMI including information on completed modification work, including: adding, removing, altering, or moving materials, tools, supplies, equipment, and fixtures in or on at least one vehicle. The CU management control logic also includes control logic for selectively reporting whether the modification work on at least one vehicle is complete or incomplete based on whether the modification objective has been achieved; and, when it is determined that the modification objective has not been achieved, employing a recommended vehicle downtime mitigation strategy to reduce downtime or idle time of at least one vehicle that has not yet been fully modified from Level 1 to Level 2 (below Level 1) by: notifying the fleet manager that for at least one vehicle whose modification work has not been completed, each vehicle has one or more potential alternative use cases; and enabling the fleet manager to select at least one of the one or more potential alternative use cases. The VDR management control logic also includes control logic for receiving commands from the HMI from the fleet manager, commanding the assignment of one or more tasks to at least one vehicle; and determining, based on data from multiple sensors, that at least one vehicle has the appropriate equipment to complete the assigned tasks. The VDR management control logic also includes control logic for activating multiple sensors and enabling them to monitor at least one vehicle and its surrounding environment; and for analyzing data from the multiple sensors and comparing the data with predetermined requirements for task allocation by executing one or more algorithms, including object detection algorithms, such as artificial intelligence (AI), machine learning (ML), and deep neural network (DNN), to determine the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle. The VDR management control logic also includes control logic for determining the daily readiness status of the vehicle based on the quantity and placement of materials, supplies, tools, fixtures, and equipment on or within at least one vehicle and the predetermined requirements for task allocation, and generating a notification to the HMI to inform the fleet manager of the daily readiness status of at least one vehicle. The VDR management control logic also includes control logic for selectively generating and transmitting to the HMI a report including a "check completed" notification or a "ready" notification when at least one vehicle meets the predetermined requirements for the assignment task; and selectively generating and transmitting to the HMI a detailed report on the assignment task readiness status of at least one vehicle when at least one vehicle does not meet the predetermined requirements for the assignment task, the report relating to the current status of materials, supplies, tools, fixtures and equipment on or within at least one vehicle.The VDR management control logic also includes control logic for generating and transmitting outputs from APD, CU management, and VDR management directly or indirectly to the HMI and cloud computing server via I / O ports, including suggested vehicle downtime mitigation strategies that can be viewed by fleet managers.

[0023] Further areas of application will become apparent from the description provided herein. It should be understood that these descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0024] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0025] Figure 1 This is a schematic diagram of a system and method for monitoring the preparation and retrofitting of autonomous vehicles for efficient fleet management, according to an exemplary embodiment.

[0026] Figure 2 This is a description based on an exemplary embodiment. Figure 1 A flowchart of the authorized personnel detection control logic for a system and method for monitoring the preparation and retrofitting of autonomous vehicles for efficient fleet management;

[0027] Figure 3 This is a description based on an exemplary embodiment. Figure 1 A flowchart of the cargo and modification control logic of a system and method for monitoring the preparation and modification of autonomous vehicles for efficient fleet management;

[0028] Figure 4 This is a description based on an exemplary embodiment. Figure 1 A flowchart of the vehicle daily preparation control logic for a system and method for monitoring the preparation and retrofitting of autonomous vehicles for efficient fleet management; and

[0029] Figure 5 This is a series of views of the human-machine interface of a system and method for monitoring the preparation and retrofitting of autonomous vehicles for efficient fleet management, according to exemplary embodiments, including... Figure 2 Description of the authorized personnel detection and control logic, Figure 3 The cargo and modification control logic, and Figure 4 The vehicle's daily preparation control logic. Detailed Implementation

[0030] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0031] refer to Figure 1The diagram schematically illustrates a system 10 for monitoring the preparation and retrofitting of an autonomous vehicle 12 for efficient fleet management. System 10 operates on vehicle 12. While vehicle 12 is shown as a truck, it should be understood that vehicle 12 can be any type of vehicle without departing from the scope or intent of this disclosure. In some examples, vehicle 12 can be a truck, bus, semi-trailer, tractor-trailer, SUV, truck, aircraft, helicopter, amphibious vehicle, boat or ship, bicycle, motorcycle, tricycle, or any other type of vehicle 12. Figure 1 The vehicle 12 shown also includes a cargo hold 14. The cargo hold 14 defines the internal volume of the vehicle 12, which is designed for storing and transporting goods 15, articles, tools, etc. It should be understood that the cargo hold 14 of various vehicle types 12 within the scope of this disclosure can vary considerably. That is, the cargo hold 14 of a semi-trailer or tractor-trailer can define a very large and substantially right-angled prism volume, relative to a cargo hold 14 defined by a bicycle or motorcycle chassis, side saddle, or the like. Similarly, the cargo hold 14 of a helicopter, aircraft, ship, or vessel can be similar in size and shape to... Figure 1 The cargo compartments of the trucks shown are significantly different.

[0032] In addition, Figure 1 In the example shown, vehicle 12 is equipped with one or more sensors 16 for detecting information about vehicle 12. Without departing from the scope or intent of this disclosure, sensor 16 may include any of a variety of sensors 16 or sensor types. In several examples, sensor 16 may include motion sensors 16, including but not limited to: microwave sensors, infrared sensors, ultrasonic sensors, vibration sensors, or one or more cameras 18, each camera 18 having a different field of view (FOV) relative to the other cameras 18. Figure 1 In this embodiment, vehicle 12 is equipped with at least one cargo compartment camera 18A and at least one passenger compartment camera 18B. Without departing from the scope or intent of this disclosure, additional cameras 18, including a surround-view camera 18C, may also be mounted on vehicle 12. The surround-view camera 18C has a field of view (FOV) extending around the vehicle 12 to which it is mounted and is positioned to collect image data about the surrounding environment of vehicle 12, including detecting obstacles, people, pets, cargo, buildings, trees, road surfaces, curbs, and any other aspects of the environment surrounding vehicle 12.

[0033] Additional sensors 14, such as an inertial measurement unit (IMU) 20 and a global positioning system (GPS) sensor 22, are also equipped on the vehicle 12. The IMU 20 measures motion, acceleration, etc., in multiple degrees of freedom. In a specific example, the IMU 20 can measure the position, motion, acceleration, etc., of the vehicle 12 in three or more degrees of freedom. The GPS sensor 22 communicates with a global positioning satellite network (not specifically shown) to determine and report the location of the GPS sensor 22 on Earth. It should be understood that the GPS sensor 22 is commonly used in navigation applications and helps determine the location of vehicles 12, packages, etc. Therefore, the GPS sensor 22 equipped on the vehicle 12 effectively reports the location of the vehicle 12 on Earth, or at least its location in relation to the global positioning satellite network, even if the vehicle 12 is not on Earth.

[0034] System 10 also includes one or more controllers 24. Controller 24 is a non-general-purpose electronic control device having a pre-programmed digital computer or processor 26 and a non-transitory computer-readable medium or memory 28 for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc. Controller 24 also includes one or more input / output (I / O) ports 30. The computer-readable medium or memory 28 includes any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, solid-state storage, optical disc (CD), digital video disc (DVD), or any other type of memory 28. The non-transitory computer-readable memory 28 does not include wired, wireless, optical, or other communication links for transmitting transient electrical or other signals. The non-transitory computer-readable memory 28 includes media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code. Processor 26 is configured to execute code or instructions. Vehicle 12 may have a controller 24, which includes a dedicated Wi-Fi controller, an engine control module, a transmission control module, a body control module, a suspension control module, a brake control module, an infotainment control module, etc. Without departing from the scope or intent of this disclosure, I / O port 30 may be configured as follows:

[0035] This disclosure includes wired communications, Wi-Fi protocols under IEEE 802.11x, cellular links, and protocols such as Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wireless Local Loop (WLL), General Packet Radio Service (GPRS), 1G, 2G, 3G, 4G Long Term Evolution (LTE), 5G, and satellite links.

[0036] The vehicle controller 24 also includes one or more application programs 32. Application program 32 is a software program configured to perform a specific function or set of functions. Application program 32 may include one or more computer programs, software components, instruction sets, processes, functions, objects, categories, instances, related data, or adaptations thereof implemented in suitable computer-readable program code. Application program 32 may be stored in memory 28 or in an attached or separate memory 28. In several aspects, application program 32 may manage powertrain functions, suspension system functions, braking system functions, aerodynamic system functions, steering system functions, and / or body control system functions in the exemplary vehicle 12. Application program 32, which manages the powertrain functions, suspension system functions, braking system functions, aerodynamic system functions, steering system functions, and body control system functions in vehicle 12, receives static and / or dynamic vehicle status information or sensor data from sensor array 16 located on vehicle 12. Therefore, sensor data may include, but is not limited to: vehicle 12 position, vehicle 12 speed, vehicle 12 acceleration, vehicle 12 cargo compartment 14 image, etc.

[0037] System 10 also includes a back office or cloud computing server 34 and may include additional infrastructure such as one or more cell towers (not specifically shown), cameras 16 (not specifically shown) installed on infrastructure such as buildings, traffic signals, etc. The cloud computing server 34 has a controller 24 with a structure substantially the same as the vehicle's onboard controller 24. That is, the cloud computing server 34 includes a processor 26, memory 28, and one or more I / O ports 30. Without departing from the scope or intent of this disclosure, the sensors of vehicle 12 communicate electronically with the onboard controller 24 and may also communicate electronically with the cloud computing server 34, directly or indirectly.

[0038] Now turn to Figure 2 And continue to refer to Figure 1 One of the applications 32 stored in the memory 28 of the vehicle controller 24 is shown in more detail. Figure 2 The application 32 shown defines a fleet management application (FMA) 36, which monitors and reports the readiness and modification monitoring status of the autonomous vehicles 12 to a fleet manager 38, thereby helping the fleet manager and / or vehicle 12 operators or drivers to effectively control one or more vehicles 12 in the fleet equipped with and operating as part of the system 10. The fleet manager 38 can be, for example, Figure 1The person shown, or fleet manager 38, can be an application 32 of fleet 12 within the automated or semi-automated management system 10. In some examples, fleet manager 38 is a person who uses one or more applications 32 (including FMA 36) to control various aspects of fleet 12 operating within system 10. Fleet manager 38 accesses FMA 36 via human-machine interface (HMI) 40, such as a desktop or laptop computer, mobile computing device (e.g., cellular phone, tablet), or other such communication and computing device. It should be understood that HMI 40 can be any of a variety of HMIs 40 without departing from the scope or intent of this disclosure. In several aspects, HMI 40 operates or otherwise supports the operation of system 10, and more specifically, at least is a fleet management user interface portion of FMA 36. In some examples, HMI 40 can communicate directly with vehicle 12 via I / O port 30 of controller 24 of vehicle 12, while in other examples, HMI 40 can communicate indirectly with vehicle 12 via I / O port 30 of cloud computing server 34. In yet another example, without departing from the scope or intent of this disclosure, HMI 40 may communicate simultaneously, sequentially, or periodically with each of the controllers 24 of vehicle 12 and cloud computing server 34.

[0039] The FMA 36 collects data from sensors 16 on vehicle 12 and infrastructure sources such as cell towers (not shown) and cameras 16 mounted on infrastructure (e.g., buildings, traffic lights, etc., not shown)). It uses the data from sensors 12 as input to the FMA 36's Authorized Personnel Detection (APD) control logic 100. The APD control logic 100 begins at block 102, where the FMA 36 determines that vehicle 12 within the fleet of managed vehicles 12 is parked. Parking detection can be performed in various ways, including but not limited to: ignition position detection, vehicle battery status, motion detection, etc. When it is determined that vehicle 12 is parked, the APD control logic 100 proceeds to block 104, where all available motion sensors 16 equipped on vehicle 12, including cargo compartment camera 18A, passenger compartment camera 18B, and surround-view camera 18C, are activated when motion is detected within the FOV of each sensor 16. When motion is detected within the FOV of one or more sensors 16, the APD control logic 100 proceeds to block 106, where camera 18 captures images and / or video of the detected motion. In one example, a person moves within the FOV of one or more sensors 16, and the APD control logic 100 records images and video of the person interacting with vehicle 12 at block 106. Subsequently, at block 108, the APD control logic 100 analyzes the captured images and video and compares the person detected within the captured images and video with profiles of known or authorized personnel. In this example, the algorithm used to analyze the captured images and video may include artificial intelligence (AI), machine learning (ML), deep neural network (DNN) algorithms, etc. In other aspects, the algorithm may utilize techniques including facial recognition, gait recognition, or other biometric information to define profiles of authorized users and compare profiles of authorized users interacting with vehicle 12 to determine whether the person interacting with vehicle 12 is an authorized user.

[0040] Fleet manager 38 identifies authorized personnel and populates a database hosted on cloud server 34 of personnel who should have access to vehicles 12 in the fleet for specific or more general purposes. That is, fleet manager 38 can determine who is authorized to load or unload vehicles 12 in the fleet but not to drive them. In contrast, others may be authorized to perform vehicle maintenance, modifications, etc., and are therefore permitted to load, unload, modify, and drive vehicles 12. Without departing from the scope or intent of this disclosure, other authorized personnel may have additional or different responsibilities towards the vehicles 12 in the managed fleet.

[0041] At box 110, once the person is determined to be an authorized personnel, APD control logic 100 proceeds to box 112, where sensor 16 continues to monitor and record information about the authorized personnel's movement and also flags any unusual, accidental, or other unauthorized behavior by the authorized personnel. In some examples, unusual or accidental behavior by the authorized personnel may include adding, removing, or even moving the location of materials or supplies within the cargo hold 14 of vehicle 12 without instruction from fleet manager 38, other applications 32 controlled by fleet manager 38, etc.

[0042] However, when it is determined at box 110 that the person is not an authorized personnel, the APD control logic 100 proceeds to box 114, whereby the APD control logic 100 causes sensor 16 to continue monitoring and recording information about the unauthorized person's movement, proactively and immediately issuing a real-time warning to fleet manager 38 about unauthorized access to vehicle 12, and preparing a detailed report including suggested response actions. In other aspects, fleet manager 38 can modify the notification and alarm settings of the APD control logic 100 so that at box 110, when an unauthorized person interacts with vehicle 12, law enforcement can be directly contacted under specific predefined circumstances.

[0043] Now turn to Figure 3 And continue to refer to Figure 1 and Figure 2 The FMA36 also includes cargo and modification (CU) control logic 200. In many fleet applications, vehicles 12 are used for a variety of different tasks, and each such task may require different tools, equipment, supplies, jigs, and other such cargo 15. In some examples, vehicles 12 are equipped with tools, equipment, jigs, supplies, or other such cargo 15 on a daily, weekly, monthly, or other such cycle. In other examples, fleet vehicles 12 are equipped with tools, equipment, supplies, jigs, or other such cargo 15 on a per-job basis. It should be understood that in some cases, fleet vehicles 12 may not be fully loaded or modified with all the tools, equipment, supplies, jigs, or other such cargo 15 necessary to perform the functions of each fleet vehicle 12 daily, or partially loaded vehicles 12 may need to perform other tasks within the fleet. Therefore, CU control logic 200 tracks and reports the cargo and modification status of vehicles 12 in the fleet to the fleet manager 38, enabling the fleet vehicles 12 to be used efficiently and effectively, regardless of their full or partial modification status.

[0044] CU control logic 200 begins at block 202, where FMA 36 determines that vehicle 12 within the fleet of managed vehicles 12 is being modified. Modification detection performed at block 202 can be accomplished in various ways, including but not limited to: fleet manager 38 selecting or directly inputting to HMI 40 to select one or more vehicles from the fleet of vehicles 12, and ignition position detection, vehicle 12 battery status, movement detection, location information, etc. When it is determined that vehicle 12 is being modified, CU control logic 200 proceeds to block 204, where sensors 16 equipped to vehicle 12 are activated to monitor the modification work. In several examples, motion sensors 16 equipped to vehicle 12 (including cargo compartment camera 18A, passenger compartment camera 18B, and surround view camera 18C) are activated and monitor the movement of people, supplies, tools, jigs, equipment, and other such cargo 15 associated with vehicle 12. At block 206, sensors 16, particularly cameras 18 of vehicle 12, capture video and images of the modification work being carried out on vehicle 12 and its surroundings. At box 208, an algorithm stored in memory 28 of controller 24 analyzes the captured images and videos to determine the modification status of vehicle 12. Without departing from the scope or intent of this disclosure, the algorithm can be any of a variety of different types of algorithms. In examples, the algorithm may include artificial intelligence (AI), machine learning (ML), deep neural network (DNN) algorithms, including object detection algorithms, etc. Generally, the algorithm compares the captured image and video data with predetermined fitting target information. Modification target information includes the quantity and placement of supplies, tools, fixtures, and equipment, and other such goods 15 on or within vehicle 12, for a specific modification purpose instructed by fleet manager 38.

[0045] The fleet manager 38 determines whether the modified vehicle 12 is intended for a specific purpose or a more general one. That is, the fleet manager 38 can determine whether vehicle 12 will be used for a specific task or a more general purpose. For example, vehicle 12 may be used solely for the transport of workers, and the modification configuration may require vehicle 12 to be equipped with optimized seating to provide the maximum permissible number of people safely transported within vehicle 12. In other examples, vehicle 12 may be used to carry piping supplies, and therefore may require the support of a certain length of pipe on shelves installed within the cargo hold 14 of vehicle 12, as well as toolboxes installed within the cargo hold 14, while leaving only space for driver and passenger seating. However, it should be understood that vehicle 12 may have other or different uses requiring different types of modifications without departing from the scope or intent of this disclosure.

[0046] At box 210, when it is determined that the modification work has been completed, CU control logic 200 proceeds to box 212, where sensor 16 continues to monitor and record information about the completed modification work, including the addition, removal, alteration, or movement of materials, tools, supplies, equipment, jigs, and other such cargo 15 within the cargo hold 14 of vehicle 12. The recorded and monitored information is reported to fleet manager 38.

[0047] However, when the modification of vehicle 12 is not complete at block 210, CU control logic 200 proceeds to block 214, where CU control logic 200 causes sensor 16 to continue monitoring and recording information about the vehicle's modification status and to generate a detailed report on the current modification status of vehicle 12. Additionally, at block 214, CU control logic 200 uses the current modification status of vehicle 12 to determine possible uses for partially or incompletely modified vehicles 12 and shares possible alternative uses with fleet manager 38. That is, CU control logic 200 includes a proposed vehicle 12 downtime mitigation strategy that reduces the idle or downtime of vehicles 12 in the fleet from a first level to a second level below the first level by ensuring that partially or incompletely modified vehicles 12 are available for alternative purposes, even when these vehicles 12 are not fully modified. In several non-limiting examples, partially modified vehicles 12 may be used in their partially modified state to transport personnel, supplies, tools, materials, equipment, fixtures, and other such goods 15, unless or until vehicle 12 has been fully and completely modified for its intended purpose within the fleet. The fleet manager is then prompted to choose one of the alternative usage scenarios, or to allow the partially or incompletely modified vehicle 12 to remain idle. The FMA36's CU control logic 200 is used both to monitor the initial installation work on the vehicles 12 in the fleet and to monitor the lifespan management of the vehicles 12. That is, the modification status of the vehicles 12 in the fleet can be continuously, intermittently, or periodically monitored throughout the lifespan of the vehicles 12 to determine whether the vehicles 12 have been properly and fully modified for their intended use. Furthermore, the vehicles 12 are monitored to determine the current modification status of the addition, removal, movement, or alteration of supplies, tools, materials, equipment, fixtures, and other such goods 15 related to the vehicles 12 during their lifespan.

[0048] Now turn to Figure 4 And continue to refer to Figures 1-3The FMA36 also includes Vehicle Daily Readiness (VDR) control logic 300. As previously described, in many fleet applications, vehicles 12 may be used for a variety of different tasks, and each such task may require the use of different tools, equipment, supplies, fixtures, and other such goods 15. Therefore, to ensure that vehicles 12 in the fleet are correctly equipped with the necessary tools, equipment, supplies, materials, fixtures, or other such goods 15 for their given tasks, the VDR control logic 300 is used to ensure that vehicles 12 assigned to a given task are properly modified for that task. The VDR control logic 300 begins at block 302, where the fleet manager 38 uses the FMA36 to assign tasks to vehicles 12 in the fleet. In a non-limiting example, without departing from the scope or intent of this disclosure, vehicles 12 in the fleet may be assigned to transport personnel, goods, supplies, tools, equipment, materials, fixtures, or other such goods 15 to different locations or for different jobs. When assigned a given task, the VDR control logic 300 of each vehicle 12 proceeds to block 304, where sensors 16 equipped to the vehicle 12 are activated to detect and monitor the required materials, cargo, supplies, tools, equipment, fixtures, or other such cargo 15 necessary to complete the assigned task. In several examples, motion sensors 16 equipped to the vehicle 12 (including cargo compartment camera 18A, passenger compartment camera 18B, and surround view camera 18C) are activated and monitor the movement of people, supplies, tools, fixtures, equipment, and other such cargo 15 associated with the vehicle 12.

[0049] At box 306, and in some non-limiting examples, sensors 16 and cameras 18 of vehicle 12 capture video and images of work being performed on vehicle 12, including the state of vehicle 12 (e.g., the presence of required supplies), including, but not limited to, materials, cargo, supplies, tools, equipment, jigs, and other such cargo 15 required for the allocation task of vehicle 12. At box 308, an algorithm stored in memory 28 of controller 24 analyzes the captured images and video to determine the presence of all required supplies in vehicle 12. Without departing from the scope or intent of this disclosure, the algorithm can be any of a variety of different types of algorithms. In examples, the algorithm may include artificial intelligence (AI), machine learning (ML), deep neural network (DNN) algorithms, etc. Generally, the algorithm compares the captured image and video data with target supply or vehicle 12 state information. Target supply or vehicle 12 state information includes the quantity and placement of supplies, tools, jigs, equipment, and other such cargo 15 for the allocation task of a particular vehicle 12.

[0050] At block 310, once it is determined that all necessary supplies are in place and vehicle 12 is otherwise suitable for the allocation purpose, VDR control logic 300 proceeds to block 312, where sensor 16 continues to monitor and record information about the status of vehicle 12 and sends a notification to fleet manager 38 via FMA 36. In several aspects, the notifications generated at block 312 include "Check Complete," "Ready," and others. Upon receiving a "Check Complete," "Ready," or other such notification, fleet manager 38 is certain that vehicle 12 is ready to perform the allocation task.

[0051] However, when the necessary supplies are not complete or fully in place in vehicle 12 at block 310, VDR control logic 300 proceeds to block 314, whereby VDR control logic 300 causes sensor 16 to continue monitoring and recording the status of vehicle 12 and to generate a detailed report on the task allocation readiness status of vehicle 12, relating to the supplies in vehicle 12 or otherwise currently equipped to vehicle 12. The current task allocation status of vehicle 12 is transmitted to fleet manager 38 via FMA 36.

[0052] It should also be understood that the APD control logic 100, CU control logic 200, and / or VDR control logic 300 can also be used to determine the current and estimated future state of vehicle 12 in relation to a crash or other incident. That is, the APD control logic 100 can be used to determine when and whether vehicle 12 has been involved in a crash, collision, etc., and will provide direct notification of the current state of vehicle 12 to the fleet manager 38 via HMI 40. Similarly, after vehicle 12 has been involved in a crash or collision that requires vehicle 12 to be repaired, the CU control logic 200 will provide the current state of vehicle 12, including repair status or repair progress status. Such repair progress or repair status information can be provided to the fleet manager 38 via HMI 40. In a further example, when vehicle 12 has been involved in a crash, collision, or other such incident, the VDR control logic 300 will present the daily readiness status of vehicle 12 via HMI 40. Even when vehicle 12 is in a partially repaired state, as long as vehicle 12 is suitable for road driving or meets certain off-road requirements, FMA36 can determine alternative tasks that can be assigned to vehicle 12 via CU control logic 200 and VDR control logic 300, regardless of the state of vehicle 12.

[0053] Now turn to Figure 5 And continue to refer to Figures 1-4 When FMA 36 sends information to fleet manager 38, vehicle 12 directly or indirectly sends data from APD control logic 100, CU control logic 200, and / or VDR control logic 300 to fleet manager 38, who then receives the information from HMI 40. For example... Figure 5 As shown, APD control logic 100 from Figure 2 Box 114 receives information including real-time notification of unauthorized personnel movement 402, video of unauthorized personnel movement 404, a status indicator 406 indicating that the video 404 is being forwarded to fleet manager 38 and / or law enforcement, and related alarms.

[0054] Similarly, at box 214, the location information 408 and modification status information of vehicle 12 from CU control logic 200 are displayed on HMI 40. FMA 36 provides location-sensitive vehicle 12 tracking, which allows fleet manager 38 to track the location of vehicles 12 within the fleet in real time. In several aspects, the location information 408 used by both FMA 36 and fleet manager 38 may include an authorized driver list 410 and the associated authorized area or range for each authorized driver in the authorized driver list 410, as shown in shaded area 412, and areas or ranges where each authorized driver in the authorized driver list 410 is not authorized to use vehicle 12, as shown in unshaded area 414. When APD control logic 100 is enabled, vehicle 12 may automatically contact one or more law enforcement agencies under predefined conditions. In a particular non-limiting example, based on the location of vehicle 12 and whether it is near the boundary of an authorized area, or any other safety-critical or security-related location, enhanced vehicle 12 tracking is used to continuously monitor and report anomalous activity of vehicle 12 and the activity of persons near vehicle 12. The system 10 sends real-time reports to the fleet manager 38, and based on the criticality of the vehicle 12's location and / or abnormal activities of people near the vehicle 12, the system 10 can automatically and immediately contact law enforcement agencies, request emergency services to be dispatched to the vehicle 12's location, etc.

[0055] Similarly, at box 314, the vehicle 12 readiness status from VDR control logic 300 is reported and displayed on HMI 40 within FMA 36. Vehicle 12 readiness status may include a vehicle safety indicator 416, which provides a clear indication of whether cargo on vehicle 12 is properly secured inside or to vehicle 12. Information used to determine whether vehicle 12 is ready to perform its assigned task and for safe modifications includes whether door locks are engaged 418, whether vehicle 12 occupants are wearing seatbelts, and whether toolbox 420 is properly secured to vehicle 12, etc. Additional similar notifications indicating that assigned vehicle 12 is ready for its task are intended to be captured within the scope of this disclosure.

[0056] The system 10 and method utilizing the FMA36 disclosed herein offer several advantages. These include providing automated incident detection and reporting using existing or retrofittable hardware, including: automatically identifying authorized users, unauthorized access, and unauthorized use; reducing vehicle 12 downtime by suggesting alternative tasks; proactively notifying fleet manager 38 about completed, ongoing, or future modifications and repairs to vehicle 12; highlighting availability limitations of vehicle 12, including modification work and components required to complete the modification work; providing location-based tracking, utilization, and reporting; and reducing the likelihood of tampering with vehicle 12 and its contents without increasing system complexity.

[0057] The descriptions in this disclosure are merely exemplary in nature, and changes that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such changes should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A system for monitoring the preparation and retrofitting of autonomous vehicles for efficient fleet management, comprising: At least one vehicle; Multiple sensors are mounted on the at least one vehicle and collect real-time information about the at least one vehicle and its surrounding environment, each of the multiple sensors having a different field of view of the at least one vehicle or its surrounding environment; A human-machine interface (HMI) is accessed by the fleet manager. Cloud computing servers; as well as The controller has a processor, memory, and one or more input / output (I / O) ports that communicate with the plurality of sensors, the HMI, and the cloud computing server. The processor executes program control logic stored in the memory, the program control logic including a fleet management application (FMA), the FMA comprising: First control logic, the first control logic is used for authorized personnel to detect APD; The second control logic is used for cargo and modified CU management; and The third control logic is used for vehicle daily preparation VDR management; The outputs from the first, second, and third control logics are transmitted directly or indirectly to the HMI and the cloud computing server via the I / O ports. The FMA generates suggested vehicle downtime mitigation strategies for the fleet manager based on the outputs from the first control logic, the second control logic, and the third control logic.

2. The system according to claim 1, wherein, The first control logic further includes: The control logic enables the fleet manager to populate a database of personnel who are authorized users of the at least one vehicle, the database being hosted on the cloud computing server.

3. The system according to claim 2, wherein, The first control logic further includes: Control logic that uses the multiple sensors to monitor the at least one vehicle and the surrounding environment of the at least one vehicle; Control logic for determining that at least one vehicle has been parked based on data from the plurality of sensors; Control logic for detecting motion within one or more fields of view of the plurality of sensors; and Control logic to determine if someone is approaching or inside the at least one vehicle.

4. The system according to claim 3, wherein, The first control logic further includes: Control logic that analyzes sensor data and compares a person approaching or inside the at least one vehicle with information on authorized users stored in a database of authorized users of the at least one vehicle; and Control logic that generates notifications to the HMI based on abnormal or unexpected behavior of authorized personnel interacting with at least one vehicle; and Control logic that proactively and immediately generates a notification to the HMI, informing the fleet manager that an unauthorized person is interacting with at least one vehicle, continuously recording the actions taken by the unauthorized person, and compiling a detailed report of the unauthorized person's actions; and Control logic that generates suggested responses in cases where authorized or unauthorized personnel use at least one of the vehicles without authorization.

5. The system according to claim 1, wherein, The second control logic also includes: Control logic that determines at least one vehicle is being modified based on data from the multiple sensors; Control logic that uses the multiple sensors to continuously monitor the at least one vehicle and its surrounding environment; and Control logic that analyzes data from the multiple sensors and compares the data from the multiple sensors with predetermined modification target information.

6. The system according to claim 5, wherein, The control logic for analyzing data from the multiple sensors also includes: One or more algorithms from the categories of artificial intelligence (AI) algorithms, machine learning (ML) algorithms, and deep neural network (DNN) algorithms, including object detection algorithms; and The quantity and placement of materials, supplies, tools, fixtures and equipment on or inside the at least one vehicle are compared with the predetermined modification objectives for each modification purpose selected by the fleet manager.

7. The system according to claim 6, wherein, The second control logic also includes: The control logic determines the modification completion status and generates a notification to the HMI, notifying the fleet manager of the modification completion status; Control logic that generates and transmits reports to the HMI, the reports including information about completed modification work, including: adding, removing, altering, or moving materials, tools, supplies, equipment, and fixtures inside or on the at least one vehicle; and Control logic that selectively reports whether the modification work of at least one vehicle is completed or not, based on whether the modification goal has been achieved.

8. The system according to claim 7, wherein, The second control logic also includes: When it is determined that the modification goal has not yet been achieved, the control logic employs the proposed vehicle downtime mitigation strategy to reduce the downtime or idle time of at least one vehicle that has not yet been fully modified from Level 1 to Level 2 (below Level 1) in the following ways: The fleet manager was notified that for at least one vehicle whose modification work had not yet been completed, each vehicle had one or more potential alternative use cases; and This allows the fleet manager to select at least one of one or more potential alternative use cases.

9. The system according to claim 1, wherein, The third control logic also includes: The control logic receives commands from the fleet manager from the HMI, the commands assigning one or more tasks to the at least one vehicle; Control logic that determines, based on data from the plurality of sensors, that the at least one vehicle has appropriate equipment to complete the assigned task; Control logic for activating the plurality of sensors and enabling the sensors to monitor at least one vehicle and the surrounding environment of the at least one vehicle; and Control logic that analyzes data from the multiple sensors and compares the data from the multiple sensors with the predetermined requirements of the assigned task.

10. The system according to claim 9, wherein, The control logic for analyzing data from the multiple sensors also includes: One or more algorithms from the categories of artificial intelligence (AI) algorithms, machine learning (ML) algorithms, and deep neural network (DNN) algorithms, including object detection algorithms; and The quantity and placement of materials, supplies, tools, fixtures and equipment on or inside the at least one vehicle are compared with the predetermined requirements of the assigned task; Based on the quantity and placement of materials, supplies, tools, fixtures and equipment on or inside the at least one vehicle, and the control logic of the predetermined requirements of the assigned task, the daily readiness status of the vehicle is determined and a notification is generated to the HMI to notify the fleet manager of the daily readiness status of the at least one vehicle. When at least one vehicle meets the predetermined requirements of the assigned task, selectively generated control logic is transmitted to the HMI, including reports of "check complete" or "ready" notifications; and When the at least one vehicle does not meet the predetermined requirements of the assignment task, control logic selectively generates and transmits a detailed report on the assignment task preparation status of the at least one vehicle to the HMI, the report relating to the current status of materials, supplies, tools, fixtures and equipment on or within the at least one vehicle.

Images