Method for monitoring blind area of a two-wheeled vehicle using an intelligent helmet for two-wheeled vehicle riders

Abstract

The present application discloses a method for monitoring at least one blind spot of a two-wheeler using a smart helmet for two-wheeler riders, the method comprising the steps of: (a) if sensor information is obtained from sensors mounted on the smart helmet, a blind spot monitoring device obtains the position of the two-wheeler, the traveling direction of the two-wheeler, and the traveling speed of the two-wheeler by referring to the sensor information; and (b) the blind spot monitoring device sends (i) rider blind spot information obtained by referring to at least one sensing angle of the perspective of the rider wearing the smart helmet and a camera or radar sensor mounted on the smart helmet, and (ii) the position of the two-wheeler, the traveling direction of the two-wheeler, and the traveling speed of the two-wheeler to at least one nearby smart helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, thereby performing at least one of a plurality of processes.

Description

[0001] This application is a divisional application of the application with the application number CN202010073407.2, the application date of January 22, 2020, and the invention title "Method and Device for Monitoring Blind Spots of Bicycles Using Smart Helmets". Technical Field

[0002] This application relates to a method and device for monitoring blind spots of a bike using a smart helmet worn by a bike rider. More specifically, it relates to a method and device for monitoring blind spots using video images obtained from a smart helmet worn by a bike rider. Background Art

[0003] Compared with the driver of an autonomous vehicle, a bike rider pays more attention to safety. In addition, when riding a bike, due to the open space where the rider is located, standard safety systems such as airbags will not work.

[0004] Therefore, bike riders must be careful to avoid traffic accidents. Specifically, bike riders must carefully observe the surrounding vehicles and make the drivers of the surrounding vehicles see the bike. Especially when parking, there will be a phenomenon where the single rear brake light of the bike is confused with one of the brake lights of the vehicle in front of the bike. In order to improve such a phenomenon, it is necessary to develop various technologies.

[0005] In addition, the perspective of a rider of a bike or motorcycle is limited, so the rider cannot confirm all the surrounding environments of the moving bike, which may lead to various accidents.

[0006] Therefore, this application provides a method for enabling a bike rider to correctly perceive the surrounding environment. Summary of the Invention

[0007] The purpose of this application is to solve all the above problems.

[0008] Another purpose of this application is to enable a rider of a moving bike to perceive the surrounding environment.

[0009] Another purpose of this application is to enable a bike rider to perceive the surrounding environment to drive the bike safely.

[0010] Another purpose of this application is to send the information obtained by a bike rider to at least one nearby vehicle through V2X communication.

[0011] As described above, in order to achieve the above purposes of this application and realize the following effects of this application, the structural features of this application are as follows.

[0012] According to an embodiment of the present application, a method for monitoring at least one blind spot of a two-wheeler using a smart helmet for two-wheeler riders includes the following steps: (a) If at least one video image of at least one first blind spot corresponding to the smart helmet worn by the rider is acquired, the blind spot monitoring device instructs an object detector to detect at least one object on the video image and confirm at least one first object located in the first blind spot among the detected objects; and (b) After the blind spot monitoring device determines the smart helmet direction and the two-wheeler traveling direction by referring to sensor information obtained from at least a part of a Global Position System (GPS) sensor, an acceleration sensor, and a geomagnetic sensor installed on the smart helmet, at least one second object corresponding to the two-wheeler among the first objects and located in at least one second blind spot is confirmed by referring to the smart helmet direction and the two-wheeler traveling direction, and the second object is displayed through a head-up display installed on the smart helmet, or the fact that the second object is located in the second blind spot is indicated by a sound alert from at least one speaker installed on the smart helmet.

[0013] In one embodiment, step (b) further includes the following steps: (b1) The blind spot monitoring device sends (i) rider blind spot information obtained by referring to at least one sensing angle of the camera or radar sensor that captures the video image and the perspective of the rider wearing the smart helmet, and (ii) the position, traveling direction, and traveling speed of the two-wheeler obtained by referring to the sensor information to at least one nearby smart helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, so as to perform at least one of the following steps: (1) Instruct at least one specific nearby vehicle among the nearby vehicles located in the rider's blind spot to refer to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle to (1-a) determine the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler, and thus (1-b) issue an alarm to at least one specific nearby driver of the specific nearby vehicle; and (2) Instruct at least one specific nearby smart helmet corresponding to at least one specific nearby two-wheeler located in the rider's blind spot among the nearby smart helmets to refer to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the nearby two-wheeler position, nearby two-wheeler traveling direction, and nearby two-wheeler traveling speed obtained from the sensor information of the specific nearby smart helmet to (2-a) determine the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler, and thus (2-b) issue an alarm to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

[0014] In one embodiment, when the specific nearby driver of the specific nearby vehicle located in the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, the blind spot monitoring device instructs the specific nearby vehicle located in the rider's blind spot to stop rotating the steering wheel or vibrate the steering wheel, so as to issue an alarm to the specific nearby driver.

[0015] In one embodiment, when the specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, when it is determined that the driving plan of the autonomous vehicle is to move towards the nearby front area of the two-wheeler by referring to (i) the position of the two-wheeler, the driving direction of the two-wheeler, and the driving speed of the two-wheeler, and (ii) the position of the autonomous vehicle, the driving direction of the autonomous vehicle, and the driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, the blind spot monitoring device instructs the autonomous vehicle located in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan.

[0016] In one embodiment, in the step (b), after the blind spot monitoring device calculates the angle difference between the smart helmet direction and the driving direction of the two-wheeler, it uses the angle difference to transform at least one first position of the first object into at least one relative position corresponding to the driving direction of the two-wheeler, and determines at least a part of the first object corresponding to at least a part of the relative position matching the second blind spot as the second object.

[0017] In one embodiment, in the step (a), the blind spot monitoring device instructs the object detector to (i) input the video image into a convolutional layer, so that the convolutional layer generates at least one feature map by performing a convolutional operation on the video image, (ii) input the feature map into an RPN, so that the RPN generates at least one proposal box corresponding to the object on the feature map, (iii) input the feature map into a pooling layer, so that the pooling layer performs a pooling operation on at least one region corresponding to the proposal box on the feature map to generate at least one feature vector, (iv) input the feature vector into an FC layer, so that the FC layer performs an FC operation on the feature vector, and (v) input the outputs from the FC layer into a classification layer and a regression layer respectively, so that the classification layer and the regression layer respectively output the classification information and regression information of each object corresponding to each proposal box, thereby detecting the object on the video image.

[0018] In one embodiment, the object detector performs the following process through a learning device to reach a learned state, and the process includes: (i) inputting a training image into the convolutional layer, enabling the convolutional layer to perform the convolutional operation on the training image to generate a feature map for learning; (ii) inputting the feature map for learning into the RPN, enabling the RPN to generate at least one proposed box for learning corresponding to at least one object for learning on the feature map for learning; (iii) inputting the feature map for learning into the pooling layer, enabling the pooling layer to perform the pooling operation on at least one region corresponding to the proposed box for learning on the feature map for learning to generate a feature vector for learning; (iv) inputting the feature vector for learning into the FC layer, enabling the FC layer to perform the FC operation on the feature vector for learning; (v) respectively inputting at least one output from the FC layer into the classification layer and the regression layer, and respectively outputting the classification information for learning and the regression information for learning of each object for learning corresponding to the proposed box for learning; and (vi) enabling the loss layer to calculate at least one loss with reference to the classification information for learning, the regression information for learning, and the GT respectively corresponding thereto, and in order to backpropagate the loss to minimize the loss, updating at least one of at least one parameter in the FC layer and the convolutional layer.

[0019] According to another embodiment of the present application, a method for monitoring at least one blind spot of a two-wheeler using a smart helmet for two-wheeler riders includes the following steps: (a) If sensor information is obtained from at least a part of a GPS sensor, an acceleration sensor, and a geomagnetic sensor installed on the smart helmet worn by the rider, the blind spot monitoring device obtains the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed by referring to the sensor information; and (b) The blind spot monitoring device sends (i) rider blind spot information obtained by referring to at least one sensing angle of the perspective of the rider wearing the smart helmet and a camera or a radar sensor installed on the smart helmet, and (ii) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed to at least one nearby smart helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, so as to perform at least one of the following processes: (1) Instructing at least one specific nearby vehicle located in the rider's blind spot among the nearby vehicles to refer to (i) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed, and (ii) the vehicle position, the vehicle driving direction, and the vehicle driving speed obtained from the sensor information of the specific nearby vehicle to (1-a) judge the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler, and thus (1-b) issuing an alarm to at least one specific nearby driver of the specific nearby vehicle; and (2) Instructing at least one specific nearby smart helmet corresponding to at least one specific nearby two-wheeler located in the rider's blind spot among the nearby smart helmets to refer to (i) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed, and (ii) the nearby two-wheeler position, the nearby two-wheeler driving direction, and the nearby two-wheeler driving speed obtained from the sensor information of the specific nearby smart helmet to (2-a) judge the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler, and thus (2-b) issuing an alarm to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

[0020] In one embodiment, in step (b), when the specific nearby driver of the specific nearby vehicle located in the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed, and (ii) the vehicle position, the vehicle driving direction, and the vehicle driving speed obtained from the sensor information of the specific nearby vehicle, the blind spot monitoring device instructs the specific nearby vehicle located in the rider's blind spot to stop rotating the steering wheel or vibrate the steering wheel, so as to issue an alarm to the specific nearby driver.

[0021] In one embodiment, when the specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, when it is determined that the driving plan of the autonomous vehicle is to move towards the nearby front area of the two-wheeler by referring to (i) the position of the two-wheeler, the driving direction of the two-wheeler, and the driving speed of the two-wheeler, and (ii) the position of the autonomous vehicle, the driving direction of the autonomous vehicle, and the driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, the blind spot monitoring device instructs the autonomous vehicle located in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan.

[0022] In another embodiment of the present application, a blind spot monitoring device for monitoring at least one blind spot of a two-wheeler using a smart helmet for a two-wheeler rider includes: at least one memory for storing instructions; and at least one processor for executing the instructions, and the processor performs the following processes: (I) If at least one video image of at least one first blind spot corresponding to the smart helmet worn by the rider is obtained, it instructs an object detector to detect at least one object on the video image, and confirms at least one first object located in the first blind spot among the detected objects; and (II) After determining the smart helmet direction and the two-wheeler driving direction by referring to the sensor information obtained from at least a part of the GPS sensor, the acceleration sensor, and the geomagnetic sensor installed on the smart helmet, at least one second object located in at least one second blind spot corresponding to the two-wheeler among the first objects is confirmed by referring to the smart helmet direction and the two-wheeler driving direction, and the second object is displayed through a head-up display installed on the smart helmet, or the second object located in the second blind spot is indicated by a sound alarm from at least one speaker installed on the smart helmet.

[0023] In one embodiment, the process (II) further includes the following process: (II-1) sending (i) rider blind spot information obtained by referring to the perspective of the rider wearing the smart helmet and at least one sensing angle of a camera or a radar sensor that captures the video image, and (ii) the position, traveling direction, and traveling speed of the two-wheeler obtained by referring to the sensor information to at least one nearby smart helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, so as to perform at least one of the following processes: (1) instructing at least one specific nearby vehicle among the nearby vehicles located in the rider blind spot to refer to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle to (1-a) determine the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler, and thereby (1-b) issue an alarm to at least one specific nearby driver of the specific nearby vehicle; and (2) instructing at least one specific nearby smart helmet corresponding to at least one specific nearby two-wheeler located in the rider blind spot among the nearby smart helmets to refer to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the nearby two-wheeler position, nearby two-wheeler traveling direction, and nearby two-wheeler traveling speed obtained from the sensor information of the specific nearby smart helmet to (2-a) determine the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler, and thereby (2-b) issue an alarm to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

[0024] In one embodiment, when the specific nearby driver of the specific nearby vehicle located in the rider blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, the processor instructs the specific nearby vehicle located in the rider blind spot to stop the rotation of the steering wheel or vibrate the steering wheel, so as to issue an alarm to the specific nearby driver.

[0025] In one embodiment, when the specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, the processor instructs the autonomous vehicle located in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan when it is determined that the driving plan of the autonomous vehicle is to move towards the nearby front area of the two-wheeler by referring to (i) the position of the two-wheeler, the driving direction of the two-wheeler, and the driving speed of the two-wheeler, and (ii) the position of the autonomous vehicle, the driving direction of the autonomous vehicle, and the driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle.

[0026] In one embodiment, in the process (II), after the processor calculates the angle difference between the intelligent helmet direction and the driving direction of the two-wheeler, the processor uses the angle difference to transform at least one first position of the first object into at least one relative position corresponding to the driving direction of the two-wheeler, and determines at least a part of the first object corresponding to at least a part of the relative position matching the second blind spot as the second object.

[0027] In one embodiment, in the process (I), the processor instructs the object detector to (i) input the video image into a convolutional layer so that the convolutional layer generates at least one feature map by performing a convolutional operation on the video image, (ii) input the feature map into the RPN so that the RPN generates at least one proposal box corresponding to the object on the feature map, (iii) input the feature map into a pooling layer so that the pooling layer performs a pooling operation on at least one region corresponding to the proposal box on the feature map to generate at least one feature vector, (iv) input the feature vector into a FC layer so that the FC layer performs a FC operation on the feature vector, and (v) input the outputs from the FC layer into a classification layer and a regression layer respectively, so that the classification layer and the regression layer respectively output classification information and regression information of each object corresponding to each proposal box, thereby detecting the object on the video image.

[0028] In one embodiment, the object detector performs the following process through a learning device to reach a learned state. The process includes: (i) inputting a training image into the convolutional layer, causing the convolutional layer to perform the convolutional operation on the training image to generate a learning feature map; (ii) inputting the learning feature map into the RPN, causing the RPN to generate at least one learning proposal box corresponding to at least one learning object on the learning feature map; (iii) inputting the learning feature map into the pooling layer, causing the pooling layer to perform the pooling operation on at least one region corresponding to the learning proposal box on the learning feature map to generate a learning feature vector; (iv) inputting the learning feature vector into the FC layer, causing the FC layer to perform the FC operation on the learning feature vector; (v) inputting at least one learning output from the FC layer into the classification layer and the regression layer respectively, and outputting learning classification information and learning regression information of each learning object corresponding to the learning proposal box respectively; and (vi) causing the loss layer to calculate at least one loss with reference to the learning classification information, the learning regression information and the GT corresponding to them respectively, and in order to backpropagate the loss to minimize the loss, updating at least one of at least one parameter in the FC layer and the convolutional layer.

[0029] According to another embodiment of the present application, a blind spot monitoring device for monitoring at least one blind spot of a two-wheeler using an intelligent helmet for two-wheeler riders includes: at least one memory for storing instructions; and at least one processor for executing the instructions, and the processor performs the following processes: (I) If sensor information is obtained from at least a part of a GPS sensor, an acceleration sensor, and a geomagnetic sensor installed on the intelligent helmet worn by the rider, the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed are obtained by referring to the sensor information; and (II) The rider blind spot information obtained by referring to (i) the perspective of the rider wearing the intelligent helmet and at least one sensing angle of a camera or a radar sensor installed on the intelligent helmet, and (ii) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed are sent to at least one nearby intelligent helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, so as to perform at least one of the following processes: (1) Instruct at least one specific nearby vehicle among the nearby vehicles located in the rider's blind spot to refer to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed, and (ii) the vehicle position, the vehicle traveling direction, and the vehicle traveling speed obtained from the sensor information of the specific nearby vehicle to (1-a) judge the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler, and thus (1-b) issue an alarm to at least one specific nearby driver of the specific nearby vehicle; and (2) Instruct at least one specific nearby intelligent helmet corresponding to at least one specific nearby two-wheeler among the nearby intelligent helmets located in the rider's blind spot to refer to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed, and (ii) the nearby two-wheeler position, the nearby two-wheeler traveling direction, and the nearby two-wheeler traveling speed obtained from the sensor information of the specific nearby intelligent helmet to (2-a) judge the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler, and thus (2-b) issue an alarm to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

[0030] In one embodiment, in the process (II), when the specific nearby driver of the specific nearby vehicle located in the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed, and (ii) the vehicle position, the vehicle traveling direction, and the vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, the processor instructs the specific nearby vehicle located in the rider's blind spot to stop rotating the steering wheel or vibrate the steering wheel, so as to issue an alarm to the specific nearby driver.

[0031] In one embodiment, when the specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, the processor instructs the autonomous vehicle located in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan when it is determined that the driving plan of the autonomous vehicle is to move towards the nearby front area of the two-wheeler by referring to (i) the position of the two-wheeler, the driving direction of the two-wheeler, and the driving speed of the two-wheeler, and (ii) the position of the autonomous vehicle, the driving direction of the autonomous vehicle, and the driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle.

[0032] In addition, a computer-readable recording medium for recording a computer program to execute the method of the present application is also provided.

[0033] The effect of the present application is to prevent traffic accidents by enabling the two-wheeler rider to perceive the surrounding environment.

[0034] Another effect of the present application is to improve the driving quality of the rider by enabling the two-wheeler rider to perceive the surrounding environment.

[0035] Still another effect of the present application is to reduce traffic accidents on the road by sending the information obtained by the two-wheeler rider to nearby vehicles through V2X communication to enable the nearby vehicles to drive safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The following drawings for explaining the exemplary embodiments of the present application are only a part of the exemplary embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative labor.

[0037] Figure 1 It is a simple schematic diagram of a blind spot monitoring device according to an embodiment of the present application. The blind spot monitoring device monitors at least one blind spot of a two-wheeler by using a smart helmet for two-wheeler riders;

[0038] Figure 2 It is a simple schematic diagram of a blind spot monitoring method according to an embodiment of the present application. The blind spot monitoring method monitors the blind spot of a two-wheeler by using a smart helmet for two-wheeler riders;

[0039] Figure 3 It is a simple schematic diagram of an object detector according to an embodiment of the present application. The object detector is used to detect at least one object from at least one video image in a method of monitoring the blind spot of a two-wheeler by using a smart helmet for two-wheeler riders;

[0040] Figure 4A simplified schematic diagram of the monitoring process according to an embodiment of the present application, where the process is to monitor the blind area of a two-wheeled vehicle using a smart helmet for two-wheeled vehicle riders in a method for monitoring the blind area of a two-wheeled vehicle;

[0041] Figure 5 A simplified schematic diagram of the driving states of a two-wheeled vehicle and surrounding vehicles in a method for monitoring the blind area of a two-wheeled vehicle using a smart helmet for two-wheeled vehicle riders according to an embodiment of the present application;

[0042] Figure 6 A simplified schematic diagram of the working state of an autonomous vehicle performing autonomous driving according to an embodiment of the present application, where the vehicle performs autonomous driving by referring to the information of the blind area of a two-wheeled vehicle from a smart helmet in a method for monitoring the blind area of a two-wheeled vehicle using a smart helmet for two-wheeled vehicle riders. Detailed implementation manners

[0043] The following detailed description of the present application refers to the accompanying drawings that are exemplarily shown as specific embodiments, so as to make the purpose, technical solutions and advantages of the present application clear. These embodiments are described in detail to enable those skilled in the art to fully implement the present application.

[0044] In addition, in the description of the present application, it should be understood that terms such as "including" and other terms in its variant forms are only for adding other technical features, additional objects, components or steps. Other purposes, beneficial effects, features and other contents of the present application will be disclosed to those skilled in the art through the description of the specification and the embodiments of the present application. The following specifically illustrates the protection content of the present application through some embodiments and drawings. Of course, they are only examples and are not intended to limit the present application.

[0045] In addition, the present application covers all possible combinations of the exemplary embodiments indicated in this specification. It should be understood that although the various embodiments of the present application are different, they are not necessarily mutually exclusive. For example, without departing from the spirit and scope of the present application, a specific shape, structure or characteristic described in conjunction with one embodiment herein can be implemented within other embodiments. Additionally, it should be understood that without departing from the spirit and scope of the present application, the positions or arrangements of the respective elements within each disclosed embodiment can be modified. Therefore, the following detailed description should not be construed as restrictive, and the scope of the present application is only defined by the properly interpreted appended claims and the full scope of the equivalents given by the claims. In the drawings, throughout several aspects, like reference numerals refer to the same or similar functions.

[0046] Any image mentioned in the content of this application may include images related to any paved or unpaved road. In this case, objects on or near the road may include vehicles, people, animals, plants, buildings, flying objects (such as airplanes or drones), or any other obstacles that may appear in a road-related scene. However, the scope of this application is not limited thereto. As another example, any image mentioned in this application may include images not related to any road, such as images related to alleyways, land, oceans, lakes, rivers, mountains, forests, deserts, the sky, or any indoor space. In this case, objects in any of the above images may include vehicles, people, animals, plants, buildings, flying objects (such as airplanes or unmanned aerial vehicles), ships, amphibious airplanes or ships, or any other obstacles that may appear in a scene related to an alleyway, land, ocean, lake, river, mountain, forest, desert, sky, or any indoor space. However, the scope of this application is not limited thereto.

[0047] The title and abstract of this application provided herein are for convenience only and do not limit or interpret the scope or meaning of the embodiments.

[0048] To enable those skilled in the art of this application to easily implement it, the preferred embodiments of this application will be described in detail by referring to the accompanying drawings as follows.

[0049] Figure 1 It is a simple schematic diagram of a blind spot monitoring device according to an embodiment of this application. The blind spot monitoring device monitors at least one blind spot of a two-wheeler by using a smart helmet for two-wheeler riders. Refer to Figure 1 , the blind spot monitoring device 100 may include: a memory 110 for storing instructions (Instruction), the instructions being to monitor the blind spot of the two-wheeler by referring to sensor information obtained from at least one sensor installed on the smart helmet worn by the two-wheeler rider; and a processor 120 for performing the following steps of monitoring the blind spot of the two-wheeler by referring to the sensor information obtained from the sensor installed on the smart helmet according to the instructions stored in the memory 110. In this application, the two-wheeler may include a unicycle, a bicycle, a tricycle, a two-wheeler, a single-wheel or three-wheel motorcycle, etc. However, the scope of this application is not limited thereto.

[0050] Specifically, the blind spot monitoring device 100 generally uses a combination of at least one computing device and at least one computer software to achieve the required system performance. The computing device includes, for example: a computer processor, Memory, Storage, input devices, and output devices or any other conventional computing components, electronic communication devices such as routers or switches, and electronic information storage systems such as network attached storage (NA) and storage area network (SAN). The computer software is instructions that enable the computing device to function in a specific manner.

[0051] In addition, the processor of the computing device may include hardware structures such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit), a cache memory, a data bus, etc. Additionally, the computing device may also include a software structure that executes an operating system and applications that implement specific functions.

[0052] However, this description of the computing device does not exclude an integrated device that includes any combination of a processor, a memory, a medium, or any other computing component for implementing the present application.

[0053] The following refers to Figure 2 to describe a blind spot monitoring method for a two-wheeled vehicle. The method is a blind spot monitoring method for a two-wheeled vehicle that uses the blind spot monitoring device 100 of an embodiment of the present application with the above structure to refer to the sensor information obtained from the sensors installed on the smart helmet.

[0054] First, in step S1, if at least one video image of at least one first blind spot is obtained from at least one camera or at least one radar sensor 10 installed on the smart helmet worn by the rider on the two-wheeled vehicle, the blind spot monitoring device 100 instructs the object detector to detect at least one object on the video image. At this time, the radar sensor may include a LiDAR sensor, a laser sensor, an ultrasonic sensor, etc., and as long as it is a sensor that can obtain an image corresponding to the surrounding environment by scanning the surrounding environment, it can be included therein.

[0055] As an example, refer to Figure 3, the blind spot monitoring device 100 can send video images to the object detector 150. Then, the object detector 150 inputs the video images into the convolutional layer 151, enabling the convolutional layer 151 to perform at least one convolution operation on the video images, thereby generating at least one feature map. Moreover, the object detector 150 can output at least one proposal box corresponding to at least one object on the feature map. Thereafter, the object detector 150 inputs the feature map into the pooling layer 153, enabling the pooling layer 153 to perform a pooling operation on at least one region corresponding to the proposal box on the feature map, thereby outputting at least one feature vector. Thereafter, the object detector 150 inputs the feature vector into the fully-connected (FC) layer 154, enabling the FC layer 154 to perform an FC operation on the feature vector, and can input at least one output from the FC layer 154 into the classification layer 155 and the regression layer 156 respectively, thereby generating classification information and regression information for each object corresponding to each proposal box to detect objects on the video images.

[0056] At this time, the object detector can reach the learned state according to the learning device.

[0057] That is, the learning device inputs at least one training image into the convolutional layer 151, enabling the convolutional layer 151 to generate at least one learning feature map by performing at least one convolution operation on the training image, and inputs the learning feature map into the region proposal network (RPN) 152, enabling the RPN 152 to output at least one learning proposal box corresponding to at least one learning object on the learning feature map. Then, the learning device inputs the learning feature map into the pooling layer 153, enabling the pooling layer 153 to perform a pooling operation on at least one region corresponding to the learning proposal box on the learning feature map, thereby generating at least one learning feature vector, and inputs the learning feature vector into the FC layer 154, enabling the FC layer 154 to perform an FC operation on the learning feature vector. Thereafter, the learning device inputs at least one learning output from the FC layer 154 into the classification layer 155 and the regression layer 156 respectively, thereby generating learning classification information and learning regression information for at least one learning object corresponding to the learning proposal box respectively. Moreover, the learning device enables the loss layer to generate at least one loss by referring to the learning classification information, the learning regression information, and the corresponding ground truth information (GT information), and updates at least one of at least one parameter in the FC layer 154 and the convolutional layer 151 in order to minimize the loss by using backpropagation of the loss. As a result of repeating the above steps, the learning device can learn the object detector.

[0058] Next, in step S2, the blind spot monitoring device 100 may confirm at least one first object located in the first blind spot among the objects detected by the object detector.

[0059] At this time, the first blind spot may be a blind spot corresponding to the smart helmet, and may be an area within a preset distance of the smart helmet and not visually perceivable by the lidar.

[0060] Next, in step S3, if sensor information is obtained from at least a part of the sensors 20 mounted on the smart helmet, such as a GPS sensor, an acceleration sensor, and a geomagnetic sensor, the blind spot monitoring device 100 may confirm the smart helmet direction and the two-wheeler driving direction by referring to the sensor information from at least a part of the GPS sensor, the acceleration sensor, and the geomagnetic sensor.

[0061] Thereafter, in step S4, the blind spot monitoring device 100 may confirm at least one second object located in at least one second blind spot corresponding to the two-wheeler among the first objects by referring to the smart helmet direction and the two-wheeler driving direction. At this time, the second blind spot may be an area around the two-wheeler that cannot be visually perceived by the rider on the two-wheeler.

[0062] That is, the blind spot monitoring device 100 may calculate at least one angle difference between the smart helmet direction and the two-wheeler driving direction, and then use the angle difference to transform at least one first position of the first object into at least one relative position corresponding to the two-wheeler driving direction, and determine at least a part of the first object corresponding to at least a part of the relative position matching the second blind spot as the second object.

[0063] As an example, referring to Figure 4 , at least one sensing angle 4 of a camera or a radar sensor mounted on the smart helmet 2 worn by the rider of the two-wheeler 1 may correspond to the blind spot of the smart helmet, that is, at least one rear area of the smart helmet, and the blind spot 5 of the two-wheeler 1 may be at least one preset area that cannot be visually perceived by the rider according to the driving direction of the two-wheeler 1.

[0064] At this time, the smart helmet direction of the smart helmet 2 may be changed by the rider. In this case, the first object located in the first blind spot, which is the blind spot of the smart helmet, detected by the object detector may not be the object detected from the second blind spot, which is the blind spot of the two-wheeler.

[0065] Therefore, the blind spot monitoring device 100 determines the angle between the helmet direction RD and the two-wheeler driving direction BD with the smart helmet 2 as a reference, and transforms the first position of the first object into a relative position based on the two-wheeler driving direction BD by referring to the determined angle, and determines at least a part of the first object whose relative position is in the second blind spot 5 as the second object, so that an object located in the blind spot 5 of the two-wheeler can be detected.

[0066] Next, in step S5, the blind spot monitoring device 100 displays the second object located in the second blind spot, which is the blind spot of the two-wheeler, through the head-up display installed on the smart helmet, or indicates that the second object is located in the second blind spot through an alarm emitted by at least one speaker installed on the smart helmet, so that the rider can perceive the object located in the blind spot of the two-wheeler, that is, perceive at least one pedestrian, at least one vehicle, or at least one other two-wheeler, to achieve the safe driving of the two-wheeler.

[0067] Meanwhile, in step S6, the blind spot monitoring device 100 can confirm at least one rider blind spot by referring to the perspective of the rider wearing the smart helmet and the sensing angle of the camera or radar sensor that captures the video image.

[0068] That is, by referring again to Figure 4 , the rider blind spot 6 can be determined, and the rider blind spot 6 is a state that is out of the sensing angle 4 of the camera or radar sensor installed on the smart helmet and the range of the perspective 3 of the rider wearing the smart helmet.

[0069] Therefore, in step S7, the blind spot monitoring device 100 can send (i) the rider blind spot information obtained by referring to the perspective of the rider wearing the smart helmet and the sensing angle of the camera or radar sensor that captures the video image, and (ii) the two-wheeler position, two-wheeler driving direction, and two-wheeler driving speed obtained by referring to the sensor information from the smart helmet to at least one smart helmet of at least one nearby two-wheeler. At this time, the blind spot monitoring device 100 can send the rider blind spot information, two-wheeler position, two-wheeler driving direction, and two-wheeler driving speed through V2X (Vehicle To Everything) communication.

[0070] Afterwards, a specific nearby vehicle located at a position within the rider's blind spot determines the likelihood of a traffic accident occurring between the specific nearby vehicle and the two-wheeler by referring to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed obtained from the blind spot monitoring device 100, and (ii) the vehicle position, the vehicle traveling direction, and the vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, and can use the likelihood to issue an alert to the specific nearby driver of the specific nearby vehicle. Additionally, at least one specific nearby smart helmet corresponding to at least one specific nearby two-wheeler located within the rider's blind spot in the nearby smart helmet can determine the likelihood of a traffic accident occurring between the specific nearby vehicle and the two-wheeler by referring to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed obtained from the blind spot monitoring device 100, and (ii) the nearby vehicle position, the nearby vehicle traveling direction, and the nearby vehicle traveling speed obtained from the sensor information of the specific nearby smart helmet, thereby issuing an alert to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

[0071] At this time, when the specific nearby driver of the specific nearby vehicle located within the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the two-wheeler position, the two-wheeler traveling direction, and the two-wheeler traveling speed obtained from the blind spot monitoring device 100, and (ii) the vehicle position, the vehicle traveling direction, and the vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, the specific nearby vehicle located within the rider's blind spot can issue an alert to the specific nearby driver by stopping the rotation of the steering wheel or vibrating the steering wheel.

[0072] As an example, referring to Figure 5 , the vehicle 30 traveling near the two-wheeler 1 confirms whether the vehicle 30 is located within the rider's blind spot 6 by referring to the rider's blind spot information received from the blind spot monitoring device 100. When the vehicle 30 is determined to be located within the rider's blind spot 6, if the traveling direction of the vehicle 30 is changed to an area where the rider of the two-wheeler 1 is inattentive, that is, the nearby front area of the two-wheeler that the rider cannot visually perceive, then a dangerous area 40 where the likelihood of a traffic accident exceeds a preset threshold is confirmed, and further, the vehicle 30 is interrupted from entering the dangerous area 40 to prevent a possible traffic accident between the vehicle 30 and the rider 1.

[0073] In addition, when a specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, when it is determined that the driving plan of the autonomous vehicle is to move towards the area in front of the vicinity of the two-wheeler by referring to (i) the position, driving direction, and driving speed of the two-wheeler, and (ii) the position, driving direction, and driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, the autonomous vehicle located in the rider's blind spot can be stopped from changing lanes due to the driving plan of the autonomous vehicle.

[0074] As an example, referring to Figure 6 , in step S11, the autonomous vehicle 200 can determine whether to change lanes and the lane change direction during lane change by referring to the autonomous driving plan 201 and at least one signal from at least one steering sensor 202. In step S12, the sensor information obtained from at least one position sensor and at least one speed sensor is used to refer to the position, driving direction, and driving speed of the autonomous vehicle. During the driving of the autonomous vehicle, if the rider's blind spot information, the position of the two-wheeler, the driving direction of the two-wheeler, and the driving speed of the two-wheeler are obtained through the V2X communication unit 204, it is possible to determine whether the autonomous vehicle 200 itself is located in the rider's blind spot by referring to the rider's blind spot information.

[0075] After that, in step S13, if it is determined that the autonomous vehicle 200 is located in the rider's blind spot, it is possible to determine whether the driving environment of the autonomous vehicle 200 is dangerous.

[0076] That is, when it is determined that the driving plan of the autonomous vehicle is to move from a position in the rider's blind spot to the area in front of the vicinity of the two-wheeler by referring to (i) the position, driving direction, and driving speed of the two-wheeler, and (ii) the position, driving direction, and driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, since the rider of the two-wheeler cannot perceive the autonomous vehicle 100, it can be determined that the possibility of a traffic accident occurring to the autonomous vehicle is greater than a preset threshold.

[0077] Then, the autonomous vehicle 200 makes the steering wheel difficult to operate or vibrates the steering wheel by activating the electronic steering device 205, so that the driver of the autonomous vehicle 200 perceives a dangerous situation. And the autonomous vehicle 200 can interrupt the lane change of the autonomous vehicle 200 towards the two-wheeler direction by activating the autonomous driving system 206, thereby preventing traffic accidents. In addition, the autonomous vehicle 200 can activate the alarm device 207 to issue an alarm to the driver of the autonomous vehicle and the rider of the two-wheeler using light or sound, thereby preventing traffic accidents.

[0078] In the above description, the process of the blind spot monitoring device monitoring the blind spots of the two-wheeler and the process of sending the rider's blind spot information to nearby vehicles or nearby smart helmets are described as being executed simultaneously. However, differently, only the step of the blind spot monitoring device sending the rider's blind spot information to nearby vehicles or nearby smart helmets can be executed.

[0079] That is, if sensor information is obtained from at least a part of the GPS sensor, acceleration sensor, and geomagnetic sensor installed on the smart helmet worn by the rider, the blind spot monitoring device can obtain the position, traveling direction, and traveling speed of the two-wheeler by referring to the sensor information from the smart helmet. After that, the blind spot monitoring device sends (i) the rider's blind spot information obtained by referring to the perspective of the rider wearing the smart helmet and the sensing angle of the camera or radar sensor installed on the smart helmet, and (ii) the position, traveling direction, and traveling speed of the two-wheeler to the nearby smart helmets corresponding to nearby vehicles and nearby two-wheelers, thereby executing at least one of the following steps: (1) instructing a specific nearby vehicle located in the rider's blind spot among the nearby vehicles to (1-a) determine the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, and thereby (1-b) issuing an alarm to the specific nearby driver of the specific nearby vehicle; and (2) instructing a specific nearby smart helmet corresponding to at least one specific nearby two-wheeler located in the rider's blind spot among the nearby smart helmets to (2-a) determine the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the nearby two-wheeler position, nearby two-wheeler traveling direction, and nearby two-wheeler traveling speed obtained from the sensor information of the specific nearby smart helmet, and thereby (2-b) issuing an alarm to the specific nearby rider corresponding to the specific nearby two-wheeler.

[0080] At this time, when the specific nearby driver of the specific nearby vehicle located in the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the vehicle position, vehicle traveling direction, and vehicle traveling speed obtained from the sensor information of the specific nearby vehicle, the blind spot monitoring device 100 instructs the specific nearby vehicle located in the rider's blind spot to stop rotating the steering wheel or vibrate the steering wheel to issue an alarm to the specific nearby driver.

[0081] In addition, when a specific nearby vehicle located in the rider's blind spot is an autonomous vehicle, when it is determined that the driving plan of the autonomous vehicle is to move to a position near the front of the two-wheeler by referring to (i) the position, driving direction, and driving speed of the two-wheeler, and (ii) the position, driving direction, and driving speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, the blind spot monitoring device 100 instructs the autonomous vehicle located in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan.

[0082] As described above, the present application detects the blind spot by referring to the sensor information from the camera or laser sensor installed on the smart helmet, obtains the blind spot detection result based on the generated helmet direction, and then monitors the driver's state using the human recognition state to confirm the smart helmet direction. By referring to the smart helmet direction and the two-wheeler driving direction, with the helmet direction as the reference, the blind spot detection result based on the two-wheeler driving direction transformed from the blind spot detection result is sent to the two-wheeler rider. After confirming the lidar blind spot that cannot be confirmed by either the sensing angle of the camera or lidar installed on the smart helmet or the viewing angle of the lidar wearing the smart helmet, the confirmed lidar blind spot is sent to nearby vehicles or nearby two-wheelers through V2X communication, so that the nearby vehicles or nearby two-wheelers can drive safely with reference to the rider's blind spot.

[0083] Moreover, the object of the technical solution of the present application or the part that contributes to the prior art can be implemented in the form of executable program commands by various computer means and can be recorded on a computer-readable recording medium. The computer-readable medium may include program commands, data files, and data structures individually or in combination. The program commands recorded on the medium may be components specifically designed for the content of the present application or may be used by those skilled in the computer software field. The computer-readable recording medium includes magnetic media (such as hard disks, floppy disks, and magnetic tapes), optical media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks), and hardware devices (such as ROM, RAM, and flash memory) designed to store and execute programs. The program commands include not only machine language codes compiled by compilers but also high-level codes that can be used by interpreters executable by computers. The aforementioned hardware devices can act as software modules to execute the actions of the present application, and they can perform the same operations in the opposite case. The hardware devices can be combined with memories such as ROM and RAM to store program commands and can include processors such as CPUs or GPUs to execute the commands stored in the memory, and also include a communication part for sending and receiving signals to and from external devices.

[0084] As described above, the present application has been explained by specific matters such as detailed components, limited embodiments, and drawings. Although the present application has been described by preferred embodiments, those skilled in the art will understand that various changes and modifications can be made to the scope of protection of the present application without departing from the spirit and scope of the present invention.

[0085] Therefore, the idea of the present application is not necessarily limited to the embodiments explained, and the following patent claims and all contents including equivalents or equivalent variations of the patent claims fall within the scope of the idea of the present application.

Claims

1. A method for using a smart helmet for a two-wheeler rider to monitor at least one blind spot of a two-wheeler, wherein, Including the following steps: (a) If at least one video image of at least one or more first blind spots corresponding to the smart helmet worn by the rider is obtained, the blind spot monitoring device performs the following steps: instructing an object detector to detect one or more objects on the video image, and confirming one or more first objects located in the first blind spot among the detected objects, where the first blind spot is the blind spot corresponding to the smart helmet, and the area within the preset distance of the smart helmet cannot be visually perceived; And (b) The blind spot monitoring device performs the following steps: (I)(I-1) Judging the smart helmet direction and the two-wheeler driving direction by referring to at least a part of the sensor information obtained from the global positioning system sensor, the acceleration sensor and the geomagnetic sensor installed in the smart helmet; (I-2) Calculating the angle difference between the smart helmet direction and the two-wheeler driving direction; (I-3) Using the angle difference to transform one or more first positions of the first object into one or more relative positions corresponding to the two-wheeler driving direction; And (I-4) Judging at least a part of the first object corresponding to at least a part of the relative positions matching one or more second blind spots as a second object to detect the second blind spot, where the second blind spot is an area around the two-wheeler that the rider cannot visually perceive; And (II) Sending (i) the rider blind spot information obtained by referring to at least one sensing angle of the perspective of the rider wearing the smart helmet and the camera or radar sensor that captures the video image, where the rider blind spot information includes information corresponding to the area outside the perspective and the sensing angle, and (ii) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed obtained by referring to the sensor information, to at least one nearby smart helmet corresponding to at least one nearby vehicle and at least one nearby two-wheeler, so as to perform at least one of the following processes: (1) Instructing at least one specific nearby vehicle located in the rider blind spot among the nearby vehicles to refer to (i) the two-wheeler position, the two-wheeler driving direction, and the two-wheeler driving speed, and (ii) the vehicle position, the vehicle driving direction, and the vehicle driving speed obtained from the sensor information of the specific nearby vehicle to (1-a) judge the possibility of a traffic accident occurring between the specific nearby vehicle and the two-wheeler, and thus (1-b) sending an alarm to at least one specific nearby driver of the specific nearby vehicle; And (2) indicating at least one specific nearby smart helmet corresponding to at least one specific nearby two-wheeler in the rider's blind spot in the nearby smart helmet, and referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the position, traveling direction, and traveling speed of the nearby two-wheeler obtained from the sensor information of the specific nearby smart helmet to (2-a) determine the possibility of a traffic accident occurring between the specific nearby two-wheeler and the two-wheeler, thereby (2-b) issuing an alarm to at least one specific nearby rider corresponding to the specific nearby two-wheeler.

2. The method according to claim 1, wherein In step (b), when the specific nearby driver of the specific nearby vehicle in the rider's blind spot operates the steering wheel of the specific nearby vehicle to move towards the nearby front area of the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the position, traveling direction, and traveling speed of the vehicle obtained from the sensor information of the specific nearby vehicle, the blind spot monitoring device instructs the specific nearby vehicle in the rider's blind spot to stop rotating the steering wheel or vibrate the steering wheel, thereby issuing an alarm to the specific nearby driver.

3. The method according to claim 1, wherein, When the specific nearby vehicle in the rider's blind spot is an autonomous vehicle, when it is determined that the driving plan of the autonomous vehicle is to move towards the nearby front area of the two-wheeler by referring to (i) the position, traveling direction, and traveling speed of the two-wheeler, and (ii) the position, traveling direction, and traveling speed of the autonomous vehicle obtained from the sensor information of the autonomous vehicle, the blind spot monitoring device instructs the autonomous vehicle in the rider's blind spot to stop the lane change of the autonomous vehicle caused by the driving plan.

Images