Method for providing a projected safety zone near a vehicle

Abstract

A system for providing a projected safety zone near a vehicle comprises an external projector, which is connected to a system controller and is capable of projecting an initial graphic onto a surface near the vehicle, wherein the initial graphic defines a predetermined safety zone, and a variety of onboard sensors, which are connected to the system controller and are capable of detecting the presence of a person who is within the projected safety zone, wherein the system controller is capable of providing a warning to the person.

Description

introduction

[0001] The present invention relates to a method for providing warnings to a person located within a safety zone projected onto the ground near a vehicle.

[0002] A generic procedure is essentially described in DE 10 2016 014 709 A1. Further state of the art can also be found in DE 10 2016 007 764 A1.

[0003] Vehicles are equipped with many sensors to monitor the environment around the vehicle and to issue warnings when objects come too close, etc. The current sensor systems of a vehicle are not designed or suitable to actively project adjacent to the vehicle or into its safe zone and warn a person who is within the safe zone and does not have credentials that authorize the person to be within the safe zone.

[0004] Thus, while current systems and methods serve their intended purpose, there is a need for a new and improved system and method for projecting a safety zone near a vehicle and alerting a person when the person is within the safety zone. Summary

[0005] According to the invention, a method for providing a projected safety zone in the vicinity of a vehicle is presented, which is characterized by the features of claim 1. The method comprises projecting, with an external projector connected to a system controller, a first graphic onto a surface in the vicinity of the vehicle, the first graphic defining a predetermined safety zone, detecting, with a plurality of on-board sensors connected to the system controller, the presence of a person located within the projected safety zone, and providing, with the system controller, a warning to the person.The method further comprises determining, with the system controller, whether the person has notification credentials that authorize the person to be within the projected security zone, and wherein providing, with the system controller, an alert to the person further comprises providing, with the system controller, an alert to the person if the person does not have credentials that authorize the person to be within the security zone.

[0006] In another aspect, determining with the system controller whether the person has credentials that authorize the person to be within the projected security zone further comprises accessing with the system controller via a wireless communication module digital credentials of the person, wherein the digital credentials are located in a cloud-based database and / or a personal device carried by the person.

[0007] According to another aspect, determining with the system controller whether the person has credentials that authorize the person to be within the projected security zone further comprises capturing, with a camera in communication with the system controller, images of the person and identifying, with a computer vision algorithm within the system controller, physical credentials, wherein the physical credentials include facial recognition, specific clothing, and / or a badge.

[0008] According to another aspect, the method further comprises identifying, with the system controller, threatening behavior exhibited by the person, and wherein providing, with the system controller, a warning to the person further comprises providing, with the system controller, a warning to the person if the person has credentials that authorize the person to be within the security zone and the system controller identifies threatening behavior exhibited by the person.

[0009] According to another aspect, identifying, with the system controller, threatening behavior exhibited by the person comprises receiving, with the system controller via an external microphone, verbal messages from the person and identifying, with the system controller, threatening language and / or capturing, with a camera in communication with the system controller, images of the person and identifying, with a computer vision algorithm within the system controller, threatening gestures made by the person.

[0010] In another aspect, providing, with the system controller, a warning to the person further comprises projecting, with the external projector in communication with the system controller, a second graphic onto the surface proximate the vehicle, the second graphic being color-coded and highlighting the position of the person.

[0011] According to another aspect, providing, with the system controller, a warning to the person further comprises, after projecting the second graphic, determining, with the system controller, that the person is still within the safety zone and providing, via external speakers, an audible warning to the person.

[0012] In another aspect, the method further comprises providing, with the system controller via a wireless communication module, an alert signal to a personal device carried by the person.

[0013] According to another aspect, providing, with the system controller, an alert to the person further comprises, after providing, via the external speakers, an audible alert to the person, determining, with the system controller, that the person is still within the safety zone, and providing, with the system controller, via the wireless communication module, a haptic alert signal to the personal device carried by the person.

[0014] In another aspect, the method further comprises detecting, with the plurality of on-board sensors, when an occupant within the vehicle exits the vehicle, and wherein projecting, with an external projector in communication with the system controller, the first graphic onto the surface proximate the vehicle further comprises projecting, with the external projector, the first graphic onto the surface proximate the vehicle and centered around the occupant of the vehicle.

[0015] According to another aspect, providing, with the system controller via the wireless communication module, an alert to a personal device carried by the person further comprises providing, with the system controller via the wireless communication module, an alert message to a personal device carried by the occupant, and providing, with the system controller via the wireless communication module, a haptic alert signal to a personal device carried by the person further comprises delivering, with the system controller via the wireless communication module, a haptic alert signal to a personal device carried by the occupant.

[0016] In another aspect, the method further comprises receiving, with the system controller via the wireless communication module, instructions from the occupant to issue a warning to a person within the projected safety zone, and wherein warning, with the system controller, the person further comprises warning, with the system controller, the person when instructions from the occupant are received.

[0017] Further described is a system for providing a projected safety zone in the vicinity of a vehicle. The system includes an external projector in communication with a system controller and configured to project a first graphic onto a surface in the vicinity of the vehicle, the first graphic defining a predetermined safety zone, and a plurality of on-board sensors in communication with the system controller and configured to detect the presence of a person within the projected safety zone, the system controller configured to warn the person.

[0018] According to another aspect, the system controller is further adapted to determine whether the person possesses credentials that authorize them to be within the projected security zone by accessing digital credentials of the person with the system controller via a wireless communication module, wherein the digital credentials are located in a cloud-based database and / or in a personal device carried by the person, and / or by taking images of the person with a camera connected to the system controller and identifying physical credentials with a computer vision algorithm within the system controller, wherein the physical credentials comprise facial recognition, specific clothing and / or a badge, wherein the system controller is adapted to warn the person,if the person does not have any credentials that authorize the person to stay within the security zone.

[0019] According to another aspect, the system controller is further adapted to identify threatening behavior exhibited by the person by receiving verbal messages from the person via an external microphone with the system controller and identifying threatening speech with the system controller and / or taking images of the person with a camera in communication with the system controller and identifying threatening gestures made by the person with a computer vision algorithm within the system controller, and wherein the system controller is further adapted to warn the person if the person has credentials that authorize the person to be within the security zone and the system controller identifies threatening behavior exhibited by the person.

[0020] According to another aspect, when the person is warned, the system controller is further configured to project a second graphic onto the surface proximate the vehicle with the external projector in communication with the system controller, the second graphic being color-coded and highlighting the position of the person, determine that the person is still within the safety zone after projecting the second graphic, and issue an audible warning to the person via external speakers and issue a warning signal to a personal device carried by the person via the wireless communication module, and determine that the person is still within the safety zone after issuing the audible warning and issue a haptic warning signal to the personal device carried by the person via the wireless communication module.

[0021] In another aspect, the system controller is further configured to detect with the plurality of on-board sensors when an occupant within the vehicle exits the vehicle and project the first graphic around the occupant of the vehicle, receive instructions from the occupant via the wireless communication module to provide a warning to a person within the projected safety zone, and warn the person when instructions are received from the occupant.

[0022] According to a further aspect, when a warning signal is delivered to the personal device carried by the person, the system controller is further adapted to deliver the warning signal to a personal device carried by the occupant via the wireless communication module, and when a haptic warning signal is delivered to the personal device carried by the person, the system controller is further adapted to deliver a haptic warning signal to the personal device carried by the occupant via the wireless communication module.

[0023] Further areas of application will become apparent from the description provided herein. It should be understood that the description and specific examples are for illustrative purposes only. Brief description of the drawings

[0024] The drawings described herein are for illustrative purposes only. Fig. 1 is a schematic diagram of a vehicle having a system for projecting a safety zone and warning a person located within the safety zone, according to an exemplary embodiment; Fig. 2 is a schematic diagram of the system according to an exemplary embodiment; Fig. 3 is a schematic perspective view of an occupant standing near a vehicle within a projected safety zone; Fig. Figure 4 is a schematic perspective view of a wheelchair user about to enter a vehicle projecting a safety zone; Fig. 5 is a schematic side view of a vehicle projecting a safety zone near an open door of the vehicle; Fig. 6 is a schematic perspective view of a vehicle projecting a safety zone near the vehicle within a construction site; Fig. 7 is a schematic perspective view of a vehicle, which is a heavy machinery vehicle, projecting a safety zone near the vehicle, with a person with credentials standing near the vehicle within the safety zone; Fig. 8 is a schematic perspective view of two pedestrians approaching a projected safety zone behind a vehicle that is reversing out of a parking space; Fig. 9 is an illustration of a person's hand being analyzed by the system controller for threatening behavior; Fig. 10 is a flowchart illustrating the analysis of the spatiotemporal characteristics of the person's physical attributes with the system controller; and Fig. 11 is a flowchart illustrating a method according to an exemplary embodiment of the present invention.

[0025] The figures are not necessarily to scale, and some features may be exaggerated or reduced in size, such as to show details of specific components. In some cases, well-known components, systems, materials, or methods have not been described in detail to avoid obscuring the present invention. Detailed description

[0026] The following description is merely exemplary in nature. Furthermore, no commitment is intended to any express or implied theory presented in the preceding technical field, background, brief summary, or the following detailed description. It should be understood that throughout the drawings, corresponding reference numerals designate like or corresponding parts and features.As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic, and / or processor device, individually or in any combination, including without limitation: an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory executing one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality. Although the figures shown herein illustrate an example with particular arrangements of elements, in actual embodiments, additional intervening elements, devices, features, or components may be present. It should also be understood that the figures are illustrative only and are not necessarily drawn to scale.

[0027] As used herein, the term "vehicle" is not limited to automobiles. Although the present technology is primarily described herein in connection with automobiles, the technology is not limited to automobiles. The concepts can be used in a wide variety of applications, such as in connection with aircraft, watercraft, other vehicles, and consumer electronics components.

[0028] According to an exemplary embodiment, Fig. 1 shows a vehicle 10 with an associated system 11 for providing a warning to an occupant 56 or warning an occupant 56 within a vehicle 10 to inform the occupant 56 of threatening vehicles when the vehicle 10 is at an intersection, according to various embodiments. In general, the system 11 cooperates with other systems within the vehicle 10 to display various information and infotainment content to the passenger. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially encloses the components of the vehicle 10. The body 14 and the chassis 12 together may form a frame. The front wheels 16 and the rear wheels 18 are each rotatably coupled to the chassis 12 near a corner of the body 14.

[0029] In various embodiments, the vehicle 10 is an autonomous vehicle and the system 11 is integrated into the autonomous vehicle 10. An autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is illustrated as a passenger car in the depicted embodiment; however, it should be understood that any other vehicle, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc., may also be used. In an exemplary embodiment, the vehicle 10 is equipped with what is known as a Level Four or Level Five automation system. A Level Four system represents "high automation," which refers to the level of performance or speed specific to a driving mode.Execution by an automated driving system for all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request for intervention. A Level Five system represents "full automation," which refers to full-time execution by an automated driving system for all aspects of the dynamic driving task under all road and environmental conditions that can be handled by a human driver. The novel aspects of the present invention are also applicable to non-autonomous vehicles.

[0030] As illustrated, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communications module 36. In one embodiment where the vehicle 10 is an electric vehicle, a transmission system 22 may not be present. The propulsion system 20, in various embodiments, may include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transfer power from the propulsion system 20 to the front wheels 16 and the rear wheels 18 of the vehicle according to selectable gear ratios.According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmission. The braking system 26 is configured to provide braking torque to the front wheels 16 and the rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, cable brakes, a regenerative braking system such as an electric machine, and / or other suitable braking systems. The steering system 24 influences the position of the front wheels 16 and the rear wheels 18. Although illustrated as including a steering wheel, the steering system 24 may not include a steering wheel in some embodiments contemplated by the present invention.

[0031] The sensor system 28 includes one or more measurement or sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar devices, lidar devices, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, and / or other sensors. The cameras may include two or more digital cameras spaced apart by a selected distance, wherein the two or more digital cameras are used to obtain stereoscopic images of the environment to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the environment surrounding the vehicle 10.In one exemplary embodiment, the plurality of sensing devices 40a-40n includes at least one of an engine speed sensor, an engine torque sensor, a voltage and / or current sensor for the electric drive motor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n further includes sensors for determining information about the environment around the vehicle 10, such as an ambient air temperature sensor, a barometric pressure sensor, and / or a still and / or video camera positioned to view the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40a-40n is capable of measuring distances in the environment around the vehicle 10.

[0032] In one non-limiting example, where the plurality of sensing devices 40a-40n includes a camera 38, the plurality of sensing devices 40a-40n measure distances using an image processing algorithm configured to process images from the camera and determine distances between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n includes a stereoscopic camera with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is mounted inside the vehicle 10, for example, on a headliner of the vehicle 10, with a view through the windshield of the vehicle 10.In another example, at least one of the plurality of sensing devices 40a-40n is a camera mounted outside the vehicle 10, for example, on a roof of the vehicle 10, with a view of the environment around the vehicle 10 and adapted to collect information (images) related to the environment outside the vehicle 10. It should be understood that various additional types of sensing devices, such as LiDAR sensors, ultrasonic range sensors, radar sensors, and / or time-of-flight sensors, are within the scope of the present invention. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more features or functions of the vehicle 10, such as, but not limited to, the drive system 20, the transmission system 22, the steering system 24, and the braking system 26.

[0033] The vehicle controller 34 includes at least one processor 44 and a computer-readable storage device or storage media 46. The at least one data processor 44 may be any custom or off-the-shelf processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among a plurality of processors connected to the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer-readable storage device or computer-readable storage media 46 may include volatile and non-volatile memory, for example, read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent orPersistent or non-volatile memory that can be used to store various operating variables while the at least one data processor 44 is turned off or shut down. The computer-readable storage device or computer-readable storage media 46 can be implemented using any number of known storage devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the vehicle 10.

[0034] The instructions may comprise one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions. The instructions, when executed by the at least one processor 44, receive and process signals from the sensor system 28, perform logic, calculations, methods, and / or algorithms for automatically controlling the components of the vehicle 10, and generate control signals for the actuator system 30 to automatically control the components of the vehicle 10 based on the logic, calculations, methods, and / or algorithms. Although in Fig. 1 only one controller 34 is shown, embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and that cooperate to process the sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control features or functions of the autonomous vehicle 10.

[0035] In various embodiments, one or more instructions of the vehicle controller 34 are included in a trajectory planning system and, when executed by the at least one data processor 44, generate a trajectory output that addresses kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions execute a graph-based approach with a tailored cost function to address various road scenarios, both urban and highway.

[0036] The wireless communication module 36 is configured to wirelessly communicate information to and from other remote units 48, such as other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate over a wireless local area network (WLAN) using IEEE 802.11 standards or using cellular data communication. However, additional or alternative communication methods, such as a dedicated short-range communication channel (DSRC), are also contemplated within the scope of the present invention.DSRC channels refer to short- to medium-range, one- or two-way wireless communication channels specifically designed for automotive use and a corresponding set of protocols and standards.

[0037] The vehicle controller 34 is a non-generalized electronic control device including a preprogrammed digital computer or processor, a memory or non-transitory computer-readable medium used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver (or input / output ports). A computer-readable medium includes any type of medium accessible by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of storage. A "non-transitory" computer-readable medium includes wired or wireless devices.Wired, wireless, optical, or other communication links that carry transitory electrical or other signals. A non-transitory computer-readable medium includes media on which data can be permanently stored and media on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable storage device. Computer code includes any type of program code, including source code, object code, and executable code.

[0038] Referring to Fig. 2, a schematic representation of system 11 is shown. System 11 includes a system controller 34A in communication with the plurality of sensing devices (on-board sensors) 40a-40n. System controller 34A may be the vehicle controller 34, or system controller 34A may be a separate controller in communication with vehicle controller 34. As discussed above, system controller 34A is in communication with the plurality of on-board sensors 40a-40n, which include at least one camera 38 capable of capturing images of an environment surrounding the vehicle and the object therein. In addition to the plurality of on-board sensors 40a-40n, system controller 34A is in communication with an external microphone 50, an external speaker 52, and an external projector 54.

[0039] Referring to Fig. 3, in an exemplary embodiment, the system controller 34A is configured to use the external projector 54 to project a first graphic 56 onto a surface proximate the vehicle 10. The surface proximate the vehicle 10 may be a ground surface, a parking lot, a roadway, or any other surface on which the vehicle 10 is located. The external projector 54 is of any known type suitable for projecting images onto the surface proximate the vehicle 10. The first graphic 56 defines a predetermined safety zone 58.

[0040] The safety zone 58 projected onto the ground near the vehicle 10 represents a visible area from which people should stay away. As shown in Fig. 3, an occupant 60 of the vehicle 10 may be, for example, a celebrity or politician, wherein when the occupant 60 leaves the vehicle 10, the safety zone 58 is intended to provide or represent a visual barrier, outside of which bystanders 62 (fans) should remain. In an exemplary embodiment, the system controller 34A is adapted to detect with the plurality of on-board sensors 40a-40n when the occupant 60 within the vehicle 10 leaves the vehicle 10 and to project the first graphic 56 around the occupant 60 of the vehicle 10. In another example, relating to Fig. 4, the safety zone 58 represents a visual barrier to keep bystanders or pedestrians 62 at a distance when a wheelchair occupant 60 is entering or exiting a vehicle 10. The system controller 34A projects the first graphic when the system controller 34A detects that the wheelchair occupant is about to enter or exit the vehicle 10, such as when a loading / unloading ramp / platform is extended.

[0041] Referring to Fig. 5, the safety zone 58 represents, in another example, a visual barrier to keep bystanders or pedestrians 62 (pedestrians, cyclists) away from the area immediately adjacent to a door 64 of the vehicle 10 that could pose a hazard if opened while such pedestrians 62 are passing by upon opening the door 64 of the vehicle 10. In an exemplary embodiment, the system controller 34A projects the safety zone 58 when the system controller 34A detects that an occupant 60 within the vehicle 10 is likely to open the door 64 and exit the vehicle 10. As shown in Fig. 5, the first graphic 56 may also include a text language 70 to convey a message to passersby 62 as shown, wherein the first graphic 56 includes both the safety zone 58 and the text language 70 indicating "open door."

[0042] Referring to Fig. 6 and Fig. 7, the safety zone 58 represents, in another example, a visual barrier to keep people and vehicles away from an area within or near a construction site. In yet another example relating to Fig. 8, the safety zone 58 represents a visual barrier to keep people away from an area behind the vehicle 10 when the vehicle 10 is in motion. As shown, the safety zone is projected behind the vehicle 10 to warn approaching pedestrians 62 that the vehicle may be about to back out of a parking space, posing a hazard to the approaching pedestrians 62. In an exemplary embodiment, the safety zone is activated when the system controller 34A detects that the vehicle 10 has been shifted into reverse and detects the presence of the approaching pedestrians 62.

[0043] In another exemplary embodiment, the system controller 34A will modify or change the safety zone 58 as circumstances change. For example, the safety zone 58 is activated when the vehicle 10 is shifted into reverse. Once the vehicle 10 begins to move, the system controller 34A may enlarge the safety zone 58 or change the color of the safety zone 58 to visually communicate the increased risk. For example, if the vehicle 10 is in reverse but not moving, the safety zone 58 may be projected as a yellow area, and when the vehicle 10 begins to move backward, the system controller 34A may enlarge the safety zone 58 and change the color to red. If the vehicle 10 is already moving and accelerating, the system controller 34A may also enlarge the safety zone 58 or change its color.

[0044] In an exemplary embodiment, the plurality of on-board sensors 40a-40n in communication with the system controller 34A are configured to detect the presence of a person 66 located within the projected safety zone 58, and the system controller 34A is configured to alert the person.

[0045] In an exemplary embodiment, when the person 66 is warned, the system controller 34A is further configured to project a second graphic 68 onto the surface proximate the vehicle 10 with the external projector 54 in communication with the system controller 34A, wherein the second graphic 68 is color-coded and highlights the position of the person 66. Referring again to Fig. 3, the system controller 34A detects the presence of the person 66 (person taking a photo of the occupant 60) within the projected safety zone 58. In response, the system controller 34A projects a second graphic 68 (circular highlight area) centered at the location of the person 66 via the projector 54. The second graphic 68 is intended to provide a visual indication to the person 66 that the person 66 has crossed the safety zone 58. The second graphic 68 may be color-coded, with the second graphic being yellow, for example, when the person 66 first enters the safety zone 58, but the second graphic may change from yellow to red if the person moves further within the safety zone 58 or closer to the occupant 60. If the person remains within the safety zone for more than a predetermined period of time, the second graphic 68 may change from yellow to red.The system controller 34A may further cause the second graphic to change colors in a pattern, turn on and off, or alternately become stronger / larger to more effectively attract the attention of the person 66.

[0046] In an exemplary embodiment, the system 11 includes a graphics engine 72 in communication with the system controller 34A. The graphics engine 72 uses data collected from the plurality of on-board sensors 40a-40n to determine an appropriate first graphic 56 and an appropriate second graphic 68 depending on the situation, such as how many people 66 are in the safety zone, etc.

[0047] In an exemplary embodiment, the system controller 34A is further configured to determine that the person is still within the safety zone 58 after projecting the second graphic 68 and to audibly warn the person 66 via external speakers 52. Referring again to Fig. 3, if the system controller 34A determines, after projecting the second graphic 68 using the plurality of on-board sensors 40a-40n, that the person has not moved out of the safety zone 58, the system controller 34A will issue an audible warning via the loudspeaker 52. The audible warning may include a subtle click or beep emitted by the loudspeaker 52 to attract the attention of the person 66 and inform them that the person 66 has entered the safety zone 58. If the behavior of the person 66 continues, the system controller 34A is adapted to increase the severity of the audible warning, for example, by increasing the volume of the audible warning, increasing the pitch of the audible warning, etc.Finally, if the behavior of person 66 escalates or continues beyond a predetermined period of time, system controller 34A may use speaker 52 to deliver an audible warning with verbal instructions to person 66. For example, speaker 53 may deliver an audible warning stating "Please step back" if a fan has approached a celebrity, as shown in FIG. Fig. 3, or the loudspeaker 53 may issue an audible warning saying “Stop!” to the couple approaching a vehicle reversing out of a parking space in front of them, as shown in Fig. 8 is shown.

[0048] In another exemplary embodiment, if the system controller 34 determines that the person 66 is still within the safety zone 58 after projecting the second graphic 68 and providing an audible warning to the person 66, the system controller 34A is further configured to provide a warning signal to a personal device 74 carried by the person 66 via the wireless communication module 36. Thus, if a person has entered the safety zone, the system controller 34A, after identifying the person via facial recognition, sends a signal to a mobile phone or other personal device carried by the person 66 to inform the person that they have entered the safety zone 58.

[0049] In another exemplary embodiment, the system controller 34A is further configured to determine that the person 66 is still within the safety zone 58 after issuing the audible warning and to send or deliver a haptic warning signal to the personal device 74 carried by the person 66 via the wireless communication module 36. Thus, if the person 66 has not responded to the projection of the second graphic 68, has not responded to the issuing of the audible warning, and has not responded to the warning signal sent to the personal device 74, the system controller 34A sends a signal to the personal device 74 causing the personal device to deliver a haptic warning, such as a vibration, a beep, a verbal warning, etc.

[0050] In an exemplary embodiment, when a warning signal is delivered to the personal device 74 carried by the person 66, the system controller 34A is further configured to deliver the warning signal to a personal device 76 carried by the occupant 60 via the wireless communication module 36, and when a haptic warning signal is delivered to the personal device 74 carried by the person 66, the system controller 34A is further configured to deliver the haptic warning signal to the personal device 76 carried by the occupant 60 via the wireless communication module 36.

[0051] Thus, whenever a signal is sent to the personal device 74 of person 66, a signal is sent to the personal device 76 of resident 60.

[0052] This informs the occupant 60 of the fact that such a signal has been sent, so that the occupant knows when and how a person 66 within the security zone 58 was warned, and draws the occupant's attention to the person 66. In another exemplary embodiment, when a signal is sent to the person's personal device 74, the system controller 34A can send such signals to pre-designated persons, such as security personnel or bodyguards, to warn them of the presence of the person 66.

[0053] In an exemplary embodiment, the system controller 34A is further configured to determine whether the person 66 possesses credentials authorizing them to be within the projected security zone 58. The system controller 34A is configured to warn the person 66 if the person 66 does not possess credentials authorizing the person 66 to be within the security zone 58.

[0054] The system controller 34A may determine whether the individual 66 possesses credentials by accessing digital credentials of the individual 66 via the wireless communication module 36, wherein the digital credentials are located in a cloud-based database 78 and / or the personal device 74 carried by the individual 66. The system controller 34A may further determine whether the individual 66 possesses credentials by capturing images of the individual 66 with a camera 38 in communication with the system controller 34A and identifying physical credentials with a computer vision algorithm 80 within the system controller 34A, wherein the physical credentials include facial recognition, specific clothing, and / or a badge.

[0055] Back on Fig. Referring to Figure 3, the first graphic 56, as shown, projects a security zone 58 comprising multiple layers, wherein the presence of a person in each specific layer of the security zone 58 is determined by the credentials they possess. As shown in Fig. 3, for example, the person 66 (photographer) may have the correct credentials that allow the person 66 to be in a first layer 82 of the security zone 58, but these credentials do not authorize the person 66 to approach the occupant 60 by entering a second layer 84 of the security zone 58. Consequently, when the person 66 moves from the first layer 82 of the security zone 58 to the second layer 84 of the security zone, the system controller 34A issues an alert to inform the person 66 that they are too close to the occupant 60.

[0056] Referring to Fig. 7, the first graphic 56 defines a safety zone 58 adjacent to a piece of heavy equipment or machinery 86 to keep bystanders away. The system controller 34A detects the presence of the person 66 within the safety zone 58; however, the person 66 is wearing protective equipment and color-coded clothing indicating that the person is a construction worker. The system controller 34A determines, using the camera 38 and the computer vision algorithm 80, that the person 66 has credentials authorizing them to be within the safety zone 58 and, consequently, does not issue an alert to the person 66.

[0057] In another exemplary embodiment, the system controller 34A is further configured to identify threatening behavior exhibited by the person 66 by 1) capturing, with the camera 38 in communication with the system controller 34A, images of the person 66 and identifying, with the computer vision algorithm 80 within the system controller 34A, threatening gestures of the person 66 and / or 2) receiving, with the system controller 34A via the external microphone 50, verbal messages of the person 66 and identifying, with the system controller 34A, threatening speech. Even if a person 66 has credentials that authorize the person 66 to be within the security zone 58, the system controller 34A is further adapted to issue a warning to the person 66 if the system controller 34A determines threatening behavior exhibited by the person 66.

[0058] In an exemplary embodiment, when images of the person 66 are captured with the camera 38 in communication with the system controller 34, and threatening gestures of the person 66 are identified with the computer vision algorithm 80 within the system controller 34A, the system controller 34A is configured to identify physical characteristics of the person 66. The physical characteristics include the location of the person 66, bodily actions such as whether the person 66 is walking, running, crawling, lying down, laughing, crying, reading, pushing, pulling, etc. The system controller 34A further identifies physical characteristics such as the position and dynamics of the legs, arms, and head of the person 66. The two-dimensional and three-dimensional location and dynamics of the skeleton of the person 66 are located and tracked as vector representations.The system controller 34A also tracks the movement dynamics of the person 66, such as direction, speed and acceleration, as well as the duration of any actions or actions (how long did the person 66 do this?).

[0059] The system controller 34A is adapted to analyze spatiotemporal characteristics of the physical characteristics of the person 66. The system controller 34A is adapted to determine whether any physical characteristics of the person 66 are indicative of threatening behavior by the person 66. The system controller 34A is adapted to compare the physical characteristics of the person 66 to "normal" patterns obtained from a remote database 88 of known patterns indicative of threatening behavior or from a neural network stored within the system controller 34A that utilizes machine learning to compile a set of known patterns indicative of threatening behavior.In addition to comparing current patterns with known patterns, other characteristics are also used to determine whether the movements and actions of a person 66 should be interpreted as threatening behavior. For example, if in a crowd, a person 66 exhibits a gesture or action, it may be interpreted by the system controller 34A as threatening behavior. However, if multiple people 66 within a group exhibit the same gesture or action, the likelihood of threatening behavior is higher. Time is always an important factor. The longer an action or activity lasts, the greater the likelihood that such action or activity constitutes threatening behavior. Similarly, the distance from the vehicle 10 or the occupant 60 is an important factor or circumstance.The farther a person 66 is from the vehicle 10 and / or the passenger 60, the less likely it is that any behavior of the person 66 is threatening. If the system controller 34A determines that the physical characteristics of the person 66 do not indicate threatening behavior on the part of the person 66, then the system controller 34A is adapted to analyze spatiotemporal characteristics of the physical characteristics of the person 66 that are continuously collected.

[0060] Referring to Fig. 9 and Fig. 10, in another exemplary embodiment of the present invention, when the system controller 34A analyzes spatiotemporal characteristics of the physical properties of the person 66 at 90, the system controller 34A is further configured to identify key points 94 of a hand 96 of a person 66 at 92. Referring to Fig. 9, keypoints 94 (as shown, joint locations within the hand and fingers) of the hand 96 of the person 66 are highlighted and linked to enable the system controller 34A to determine a shape or gesture displayed by the hand 96. The system 11 may utilize, as non-limiting examples, mesh model matching, depth-based methods, deep learning-based post-estimation methods such as open pose or deep pose, or multi-view bootstrapping when generating a keypoint model of the hand 96.

[0061] The system controller 34A is further adapted at 98 to compare identified keypoints 94 of the hand 96 of the person 66 with known threatening gestures stored in the remote database 88. Alternatively, the system controller 34A may compare the identified keypoints 94 of the hand 96 of the person 66 with known threatening gestures stored in a neural network stored in the system controller 34A that utilizes machine learning to compile a set of known threatening gestures. If, at 100, the system controller 34A determines that the identified keypoints 94 of the hand 96 of the person 66 match at least one of the known threatening gestures stored in the database 88, then the system controller 34A is adapted to determine that the person 66 is exhibiting threatening behavior, as indicated by line 102.If the system controller 34A determines that the identified keypoints 94 of the hand 96 of the person 66 do not match at least one of the known threatening gestures stored in the database 88, then the system controller 34A is operable to proceed to compare identified keypoints 94 of the hand 96 of the person 66 with the remote database 88 of known threatening gestures, as indicated by line 104.

[0062] In another exemplary embodiment, when system controller 34A receives verbal messages from person 66 via external microphone 50 and threatening speech is identified by system controller 34A, system controller 34A is further configured to extract features of acoustic signals captured by microphone 50 and compare features of the acoustic signals indicative of threatening behavior to an optimized model of known sounds stored in remote database 88. System controller 34A is further configured to determine that person 66 is exhibiting threatening behavior if the captured acoustic signals match known sounds from the optimized model.Verbal and non-verbal sounds that indicate threatening behavior include, but are not limited to, shouting, hitting / hitting with a stick, pushing or hitting sounds, swearing, abusive words, threatening words, and bullying and sarcastic sounds such as laughter / a mocking deep male voice.

[0063] The optimized model is created and updated in the remote database 88. It begins with a non-speech audio event dataset, which includes collected, augmented, and grouped data taking into account various sound factors such as season, weather conditions, locations, background noise and reverberation, vehicle speed, etc. The data from the non-speech audio event dataset is processed to generate a preprocessed non-speech audio event dataset. Distinguishing features from the preprocessed non-speech audio event dataset are identified and fed into a training and testing dataset. Model training / testing occurs using machine learning algorithms to optimize a candidate model, producing an optimized model for detected threatening sounds and speech.

[0064] In an exemplary embodiment, acoustic signals are captured by microphone 50, and system controller 34A extracts features of the acoustic signals, such as, but not limited to, mel-frequency cepstrum coefficients (MFCC), filter bank energies, log filter bank energies, and subband spectral centroids. If such features match features within the optimized model of known threatening sounds, system controller 34A determines that person 66 is exhibiting threatening behavior.

[0065] In yet another exemplary embodiment, the system controller 34A is further configured to receive instructions from the occupant 60, via the wireless communication module 36, to alert a person 66 within the projected safety zone 58 and to alert the person 66 when instructions are received from the occupant 60. Thus, an occupant may send instructions to the system controller 34A to issue an alert to a person 66 using an application on a personal device 76 or using a simple handheld or key fob actuator. This would override any determination by the system controller 34A that the person 66 has proper notification credentials or is otherwise permitted to be within the safety zone 58. The occupant 60 may selectively trigger the alert each time the occupant 60 deems an alert to the person necessary.

[0066] Referring to Fig. 11 includes a method 200 for providing a projected safety zone 58 proximate a vehicle 10, beginning at block 202, projecting, with an external projector 54 in communication with a system controller 34A, a first graphic 56 onto a surface proximate the vehicle 10, the first graphic 56 defining a predetermined safety zone 58, proceeding to block 204, detecting, with a plurality of on-board sensors 40a-40n in communication with the system controller 34A, the presence of a person 66 located within the projected safety zone 58, and proceeding to block 206, providing an alert to the person 66 with the system controller 34A.

[0067] In an exemplary embodiment, the method 200 further includes, proceeding to block 208, determining, with the system controller 34A, whether the person 66 has credentials that authorize the person 66 to be within the projected security zone 58, and providing, with the system controller 34A, a warning to the person 66 in block 206 further includes providing, with the system controller 34A, a warning to the person 66 if the person 66 does not have credentials that authorize the person 66 to be within the security zone 58, as indicated by a line 210.

[0068] In an exemplary embodiment, determining with the system controller 34A whether the person 66 has credentials that authorize the person 66 to be in the projected security zone 58 at block 208 further comprises accessing with the system controller 34A via a wireless communication module 36 digital credentials of the person 66, wherein the digital credentials are located in a cloud-based database 78 and / or in a personal device 74 carried by the person 66.

[0069] In another exemplary embodiment, determining, with the system controller 34A, whether the person 66 has credentials that authorize the person 66 to be in the projected security zone 58 in block 208 further comprises capturing, with a camera 38 in communication with the system controller 34A, images of the person 66 and identifying, with a computer vision algorithm 80 in the system controller 34A, physical credentials, wherein the physical credentials include facial recognition, specific clothing, and / or a badge.

[0070] In an exemplary embodiment, the method 200 further includes, transitioning from block 208 to block 212, identifying, with the system controller 34A, threatening behavior exhibited by the person 66, wherein providing, with the system controller 34A, a warning to the person 66 in block 206 further includes providing, with the system controller 34A, a warning to the person 66 if the person 66 has credentials authorizing the person 66 to be within the security zone 58, as indicated by a line 214, and the system controller 34A identifies threatening behavior exhibited by the person 66, as indicated by a line 216.

[0071] In an exemplary embodiment, identifying, with the system controller 34A, threatening behavior exhibited by the person 66 in block 212 further comprises receiving, with the system controller 34A via an external microphone 50, verbal messages from the person 66 and identifying, with the system controller 34A, threatening speech and / or capturing, with a camera 38 in communication with the system controller 34A, images of the person 66 and identifying, with a computer vision algorithm 80 within the system controller 34A, threatening gestures made by the person 66.

[0072] In another exemplary embodiment, providing, with the system controller 34A, a warning to the person 66 in block 206 further comprises, transitioning to block 216, projecting, with the external projector 54 in communication with the system controller 34A, a second graphic 68 onto the surface proximate the vehicle 10, wherein the second graphic 68 is color-coded and highlights the position of the person 66.

[0073] In an exemplary embodiment, providing, with the system controller 34A, a warning to the person 66 in block 206 further comprises, after projecting the second graphic 68 in block 218, proceeding to block 220, determining, with the system controller 34A, that the person 66 is still within the safety zone 58, as indicated by a line 222, and, proceeding to block 224, providing, via external speakers 52, an audible warning to the person 66. If, in block 220, the person 66 is no longer within the safety zone 58, the method 200 returns to block 204, as indicated by a line 226.

[0074] In another exemplary embodiment, the method 200 further includes, transitioning from block 224 to block 228, providing, with the system controller 34A via a wireless communication module 36, an alert signal to a personal device 74 carried by the person 66.

[0075] In an exemplary embodiment, providing, with the system controller 34A, an alert to the person 66 in block 206 further comprises, after delivering, via the external speakers 52, an audible alert to the person 66 in block 224, proceeding to block 230, determining, with the system controller 34A, that the person 66 is still within the safety zone 58, and, proceeding to block 232, providing, or delivering, with the system controller 34A, via the wireless communication module 36, a haptic alert signal to the personal device 74 carried by the person 66. If, in block 230, the system controller 34A determines that the person 66 is no longer within the safety zone 58, the method 200 returns to block 204, as indicated by a line 226. After delivering the haptic signal in block 232, the method 200 returns to block 204, as indicated by line 226.

[0076] In another exemplary embodiment, the method 200 further comprises, in block 234, detecting, with the plurality of on-board sensors 40a-40n, when an occupant 60 within the vehicle 10 exits the vehicle 10, wherein projecting, with the external projector 54 in communication with the system controller 34A, the first graphic 56 onto the surface proximate the vehicle 10 in block 202 further comprises projecting, with the external projector 54, the first graphic 56 onto the surface proximate the vehicle 10 and centered around the occupant 60 of the vehicle 10.

[0077] In an exemplary embodiment, providing, with the system controller 34A via the wireless communication module 36, an alert to a personal device 74 carried by the person 66 in block 228 further comprises providing, with the system controller 34A via the wireless communication module 36, an alert signal to a personal device 76 carried by the occupant 60, and providing, with the system controller 34A via the wireless communication module 36, a haptic alert signal to a personal device 74 carried by the person 66 further comprises providing, with the system controller 34A via the wireless communication module 36, a haptic alert signal to a personal device 76 carried by the occupant 60.

[0078] In another exemplary embodiment, if the system controller 34A does not identify any threatening behavior exhibited by the person 66 in block 212, then, proceeding to block 238 as indicated by line 236, the method 200 further comprises receiving, with the system controller 34A via the wireless communication module 36, instructions from the occupant 60 to provide a warning to a person 66 within the projected safety zone 58, wherein providing, with the system controller 34A, an alert to the person 66 in block 206 further comprises providing, with the system controller 34A, an alert to the person 66 when instructions are received from the occupant 60, as indicated at line 240. If no instructions are received from the occupant 60 at block 238, the method 200 returns to block 204, as indicated by line 242.

[0079] The method of the present invention offers the advantage of warning a person when the person is within a safety zone.

Claims

[1] A method for providing a projected safety zone (58) in the vicinity of a vehicle (10), comprising: Projecting, with an external projector (54) connected to a system controller (34A), a first graphic (56) onto a surface proximate the vehicle (10), the first graphic (56) defining a predetermined safety zone (58); Detecting, with a plurality of on-board sensors (40a-40n) in communication with the system controller (34A), the presence of a person (66) located within the projected safety zone (58); and Providing, with the system controller (34A), a warning to the person (66); characterized by : Determining, with the system controller (34A), whether the person (66) has credentials that authorize the person (66) to be within the projected security zone (58); and wherein providing, with the system controller (34A), a warning to the person (66) further comprises providing, with the system controller (34A), a warning to the person (66) if the person (66) does not have credentials authorizing the person (66) to be within the security zone (58). [2] The method of claim 2, wherein determining, with the system controller (34A), whether the person (66) has credentials that authorize the person (66) to be within the projected security zone (58) further comprises: Accessing, with the system controller (34A) via a wireless communication module (36), digital credentials of the person (66), wherein the digital credentials are located in a cloud-based database (78) and / or a personal device (76) carried by the person (66). [3] The method of claim 1, wherein determining, with the system controller (34A), whether the person (66) has credentials that authorize the person (66) to be within the projected security zone (58) further comprises: Recording, with a camera (38) connected to the system controller (34A), images of the person (66); and Identifying, with a computer vision algorithm within the system controller (34A), physical credentials, wherein the physical credentials include facial recognition, specific clothing, and / or a badge. [4] The method of claim 1, further comprising: Identifying, with the system controller (34A), threatening behavior exhibited by the person (66); and wherein providing, with the system controller (34A), a warning to the person (66) further comprises providing, with the system controller (34A), a warning to the person (66) if the person (66) has credentials authorizing the person (66) to be within the security zone (58) and the system controller (34A) identifies threatening behavior exhibited by the person (66). [5] The method of claim 4, wherein identifying, with the system controller (34A), threatening behavior exhibited by the person (66) comprises: Receiving, with the system controller (34A) via an external microphone, verbal messages from the person (66) and identifying, with the system controller (34A), a threatening language; and / or Capturing, with a camera (38) in communication with the system controller (34A), images of the person (66) and identifying, with a computer vision algorithm within the system controller (34A), threatening gestures made by the person (66). [6] The method of claim 1, wherein providing, with the system controller (34A), a warning to the person (66) further comprises projecting, with the external projector (54) in communication with the system controller (34A), a second graphic (56) onto the surface proximate the vehicle (10), the second graphic (56) being color-coded and highlighting the position of the person. [7] The method of claim 6, wherein providing, with the system controller (34A), a warning to the person further comprises: after projecting the second graphic (56), determining, with the system controller (34A), that the person (66) is still within the safety zone (58); and Providing, via external loudspeakers (52), an acoustic warning to the person (66). [8] The method of claim 7, further comprising delivering, with the system controller (34A) via a wireless communication module (36), a warning signal to a personal device (76) carried by the person (66). [9] The method of claim 8, wherein providing, with the system controller (34A), a warning to the person (66) further comprises: after providing, via the external loudspeakers (52), an audible warning to the person (66), determining, with the system controller (34A), that the person (66) is still within the safety zone (58); and Delivering, with the system controller (34A) via the wireless communication module (36), a haptic warning signal to the personal device (76) carried by the person (66).

Images