Method and device for carrying out early warning on vehicle, electronic equipment and computer readable storage medium

Abstract

The invention provides an early warning method and device for a vehicle, electronic equipment, a computer readable storage medium and a computer program product. The vehicle early warning method comprises the following steps: capturing one or more images of a vehicle during driving through a sensor; performing target detection for each of the one or more images to obtain one or more bounding boxes of one or more objects and a category of the one or more objects; and determining whether to perform early warning on the vehicle based on the one or more bounding boxes and the categories of the one or more objects.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method and device for early warning of a vehicle, an electronic device, and a computer-readable storage medium. Background Art

[0002] As the number of vehicles increases, the probability of traffic accidents between different types of vehicles is also on the rise. For example, a non-motor vehicle driver may not pay attention to the vehicle ahead and may collide with the door of a motor vehicle, thus causing a traffic accident.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of the present disclosure, a method for issuing an early warning to a vehicle is provided, comprising: capturing one or more images during driving of the vehicle via a sensor; performing target detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and one or more categories of the objects; and determining whether to issue an early warning to the vehicle based on the one or more bounding boxes and the one or more categories of the objects.

[0005] According to another aspect of the present disclosure, a device for providing an early warning for a vehicle is provided, comprising: a capture unit configured to capture one or more images of the vehicle during driving via a sensor; a target detection unit configured to perform target detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and one or more categories of the objects; and an early warning unit configured to determine whether to provide an early warning for the vehicle based on the one or more bounding boxes and the one or more categories of the objects.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the above-mentioned method for providing early warning to a vehicle.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned method for providing early warning to a vehicle.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned method for providing early warning to a vehicle are implemented.

[0009] Further features and advantages of the present disclosure will become apparent from the exemplary embodiments described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0011] Figure 1 is a flow chart showing a method for warning a vehicle according to an exemplary embodiment;

[0012] Figure 2 is a schematic diagram illustrating model distillation according to an exemplary embodiment;

[0013] Figure 3 is a flow chart showing a method for early warning of a vehicle according to another exemplary embodiment;

[0014] Figure 4 is a flowchart showing matching each other vehicle with an open door and each open door according to determining that the category of one or more objects includes other vehicles with open doors and open doors according to an exemplary embodiment;

[0015] Figure 5 is a flowchart showing a method of determining whether to warn a vehicle based on one or more bounding boxes and one or more object categories according to an exemplary embodiment;

[0016] Figure 6 is a schematic diagram showing correction of a preset warning area according to an exemplary embodiment;

[0017] Figure 7 is a structural block diagram showing a device for providing early warning to a vehicle according to an embodiment of the present disclosure;

[0018] Figure 8 is a block diagram showing a structure of a computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0021] As the number of vehicles increases, the probability of traffic accidents between different types of vehicles is also on the rise. For example, a non-car driver may not pay attention to the vehicle ahead and may collide with the door of a car, causing a traffic accident. In response to this situation, early warning systems are being installed on vehicles to provide safety warnings to drivers by detecting moving objects around the vehicle in real time.

[0022] On the one hand, a door opening warning system can be installed on the car. When the driver and passengers are about to open the door when the car is parked, the door opening warning system can monitor the objects in the blind spot behind the car in real time through ranging sensors such as radar or ultrasonic waves. If the door opening warning system detects that there may be a risk of collision due to the door opening, it will warn the driver and passengers and indicate the source of the collision risk to the driver and passengers.

[0023] On the other hand, a collision warning system can be installed on non-automobiles (for example, two-wheeled vehicles such as motorcycles, electric bicycles, two-wheeled balancing vehicles, electric scooters, and unicycles such as single-wheeled balancing vehicles). During driving, the object in front can be detected by a ranging sensor such as radar or ultrasonic wave, and then a warning is given to the driver.

[0024] However, the inventors have discovered that not all cars are equipped with door opening warning systems, and even if they are installed, the warning accuracy is not high; and the collision warning systems not installed on cars are mainly based on ranging sensors such as radar or ultrasound to detect the distance of moving objects, and are unable to detect objects such as open doors, which will also make the accuracy of the collision warning systems insufficient.

[0025] In order to solve the above technical problems, the present disclosure provides a new method and device for warning a vehicle, an electronic device, a computer-readable storage medium and a computer program product. It will be clearly seen in the detailed description below that the vehicle warning method according to the embodiment of the present disclosure can capture one or more images during the driving of the vehicle, locate and identify objects in front of the vehicle and classify objects based on such visual information, and provide a safety warning to the driver based on the position and category of one or more objects. Compared with traditional ranging sensors, warnings based on target detection, positioning and classification of image visual information will be more efficient and accurate. At the same time, since the vehicle according to the embodiment of the present disclosure can only use one image sensor as a target detection sensor, the system can be simplified, the cost can be reduced, the use is easy and the failure rate is low.

[0026] An exemplary embodiment of the vehicle warning method disclosed herein will be further described below with reference to the accompanying drawings.

[0027] Figure 1 FIG. 1 is a flow chart of a vehicle warning method 100 according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method 100 for issuing an early warning to a vehicle may include: step S110, capturing one or more images of the vehicle during driving via a sensor; step S120, performing target detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and one or more categories of the objects; and step S130, determining whether to issue an early warning to the vehicle based on the one or more bounding boxes and the one or more categories of the objects.

[0028] Compared with the traditional ranging sensor, the vehicle warning method according to the embodiment of the present disclosure can capture one or more images during the vehicle driving based on the visual sensor, and locate and identify the object in front of the vehicle and classify the object based on the image visual information. Since the warning based on the target detection, positioning and classification of the image visual information is more efficient, and only a small number of image sensors (for example, one) can be used to achieve the above operations, the system can be simplified, the cost can be reduced, the use is simple and the failure rate can be reduced.

[0029] According to some embodiments of the present disclosure, the vehicle may be a two-wheeled vehicle or a unicycle, such as a motorcycle, an electric bicycle, a two-wheeled balance vehicle, an electric scooter, or a unicycle such as a single-wheeled balance vehicle.

[0030] In step S110, the sensor may be any suitable sensor capable of capturing an image, such as a camera, a video camera, a webcam, etc. Such sensors may be independent devices as described above, or may be included in various electronic devices (e.g., mobile phones, computers, personal digital assistants, tablet computers, wearable devices, etc.). The scope of the subject matter claimed in the present disclosure is not limited in the above aspects.

[0031] In step S110, the sensor may be located at any position of the vehicle as long as it can capture moving objects around (e.g., in front of) the vehicle in real time, such as being installed above the front wheels of the vehicle, or installed on the front storage basket of the vehicle, etc., preferably installed in the direction of the central axis of the vehicle.

[0032] According to some embodiments of the present disclosure, in step S120, performing target detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and one or more categories of the objects may include: performing target detection on each of the one or more images based on a neural network model to obtain one or more bounding boxes of one or more objects and one or more categories of the objects.

[0033] It will be understood that the neural network model can be a YOLOv6 model or other models based on deep neural network algorithms to perform target detection on each image.

[0034] According to some embodiments of the present disclosure, performing target detection on each of the one or more images based on a neural network model may include: performing target detection on the one or more images based on a student model network, wherein the student model network is obtained by the following operations: training a teacher model network, the teacher model network having a first parameter amount; and obtaining a student model network based on the teacher model network, the student model network having a second parameter amount less than the first parameter amount.

[0035] Figure 2 is a schematic diagram showing model distillation according to an exemplary embodiment. Figure 2 The target detection model 200 may include a teacher model network 210 with relatively more parameters and stronger learning ability and a student model network 220 with relatively fewer parameters and weaker learning ability. In the training phase, the teacher model network 210 is first trained using training samples, and then the error is distilled based on the following formula to obtain the student model network 220 through "knowledge transfer":

[0036]

[0037] in, is the cross entropy, p(x) is the output of the student model network, q(x) is the output of the teacher model network, l(x) is the logits output of the student model network, and z(x) is the logits output of the teacher model network.

[0038] Based on the teacher model network to guide the training of the student model network, the feature representation "knowledge distillation" learned by the complex and strong learning teacher model network can be further passed to the student model network with a small number of parameters and weak learning ability, thereby obtaining a fast and expressive student model network. Accordingly, compared with the output results obtained based only on the student model network, the accuracy of the model can be improved (for example, by 10%).

[0039] According to some other embodiments of the present disclosure, performing target detection on each of the one or more images based on a neural network model may include: reparameterizing the neural network model to couple multiple groups of parameters corresponding to multiple branches of the neural network model; and performing target detection on each of the one or more images based on the model-reparameterized neural network model.

[0040] Generally, the neural network model has a multi-branch network structure in both the training stage and the application reasoning stage, and each branch network structure is associated with a set of parameters. In the training stage, it is expected to obtain more accurate training results through a large number of training samples, while in the application reasoning stage, it is expected that the neural network model can be deployed on different platforms and the reasoning results can be obtained quickly. Therefore, the multi-branch neural network structure in the training stage can be decoupled from the multi-branch neural network structure in the reasoning stage, and the multiple sets of parameters corresponding to multiple branches in the reasoning stage can be coupled to realize single-branch operation.

[0041] For example, a neural network model at a certain layer in the inference stage may have three branches A1, A2, and A3. Among them, branch A1 can implement the following operations: Branch A2 can perform the following operations: Branch A3 can perform the following operations: After coupling the parameters of the above three branches, a coupled single branch A can be obtained, which can implement the following operations:

[0042] By coupling multiple sets of parameters of multiple branches of a certain layer of neural network model, a single-branch neural network model can be obtained and equivalent operation results can be achieved. Since the single-branch operation speed is faster, the inference results of the neural network model can be obtained more quickly.

[0043] According to some other embodiments of the present disclosure, performing target detection on each of the one or more images based on the neural network model may include edge deployment of the neural network model, that is, deploying the neural network model on the application platform side instead of on the cloud. Since there is no need to transmit data to the cloud during the application reasoning stage, latency can be reduced, response speed can be increased, and dependence on network bandwidth can be reduced, while protecting user privacy.

[0044] According to some embodiments of the present disclosure, in step S120, the category of the one or more objects may include one or more of the following: other vehicles with unopened doors, other vehicles with opened doors, and open doors.

[0045] In some examples, target detection can be performed on each image based on a neural network model. When training such a neural network model, target recognition can be performed on other vehicles with unopened doors, other vehicles with open doors, open doors, and other categories of objects based on a large number of training sample sets, but target recognition is not performed on other objects. This is because in the warning application scenario of the present disclosure, objects associated with door opening collision warnings are usually more concerned, thereby reducing the size of the training sample set and improving the accuracy of target recognition.

[0046] In some examples, the bounding boxes of other vehicles with open doors and other vehicles with open doors can be obtained based on the actual outlines of the vehicle doors (open and unopened) and the trunk door. This bounding box detection mode is conducive to making the bounding box of each other vehicle with open doors contain the bounding box of at least one open door.

[0047] In some examples, after performing target detection on an image, a category ID may be assigned to each identified object, such as other vehicle 1 with unopened doors, other vehicle 2 with unopened doors, other vehicle 1 with opened doors, other vehicle 2 with opened doors, open door 1, open door 2, etc., to facilitate subsequent matching of these objects.

[0048] It will be appreciated that if it cannot be determined whether the vehicle has an open door after performing object detection, the vehicle may be classified as another vehicle with no open doors.

[0049] It will also be understood that target detection can also be performed on each of the one or more images based on the neural network to obtain position information of one or more bounding boxes. In some examples, the position information of each bounding box can be the center point of the bottom border of the bounding box. In other examples, the position information of each bounding box can be the geometric center point of the bounding box. In yet other examples, the position information of each bounding box can be the center point of the top border of the bounding box. The position information can be, for example, the two-dimensional coordinates or three-dimensional coordinates of the center point. The position information of the bounding box can refer to the coordinate information of any suitable point of the bounding box, and the scope of the subject matter claimed for protection in the present disclosure is not limited in the above aspects.

[0050] Figure 3 FIG. 1 is a flow chart showing a vehicle warning method according to another exemplary embodiment. Figure 3 As shown, the vehicle warning method 300 may include the reference Figure 1 Steps S310-S330 similar to steps S110-S130 in the described vehicle warning method 100; and step S340, matching each other vehicle with open doors and each open door based on determining that the category of one or more objects includes other vehicles with open doors and open doors, wherein in step S330, whether to warn the vehicle can be further determined based on the matching results.

[0051] Since errors may occur in the target detected objects, for example, one or more other vehicles are mistakenly identified as other vehicles with open doors without opening their doors, one or more other vehicles have opened their doors but the corresponding open doors are not detected, or one or more open doors are detected but the corresponding other vehicles with open doors are not detected. Therefore, after performing target detection on each image and detecting these objects, they are matched one by one, and further determining whether to warn the vehicle based on the matching results in step S330, thereby effectively avoiding misdetection of other vehicles with open doors and open doors, obtaining more accurate open door information, and thereby improving the accuracy of subsequent warnings to the driver.

[0052] Figure 4 1 is a flow chart showing matching each other vehicle with an open door and each open door according to an exemplary embodiment based on determining that the category of one or more objects includes other vehicles with open doors and open doors. Figure 4In step S340, according to determining that the category of one or more objects includes other vehicles with open doors and open doors, matching each other vehicle with open doors and each open door may include, for each other vehicle with open doors and each open door: step S410, calculating a first area of ​​a corresponding bounding box of the other vehicle with open doors and a second area of ​​a corresponding bounding box of the open door; step S420, calculating an overlapping area of ​​the first area and the second area; step S430, calculating a quotient of the overlapping area and the second area as a matching detection factor. value; step S440, determining that the other vehicle with the open door matches the open door based on determining that the matching detection value exceeds a preset threshold; step S450, determining that the other vehicle with the open door does not match the open door based on determining that the matching detection value does not exceed the preset threshold; step S460, determining that the other vehicle with the open door is a misdetected vehicle based on determining that the door that is not opened matches the other vehicle with the open door; and step S470, determining that the open door is a misdetected door based on determining that the other vehicle with the open door matches the open door.

[0053] After performing object detection on each of the one or more images, other vehicles with open doors, other vehicles with unopened doors, and open doors can be identified. Then, other vehicles with open doors and open doors are selected from these objects for further matching to bind the vehicle to the door.

[0054] Specifically, the area S1 of the corresponding bounding box of each other vehicle with an open door can be calculated, and the area S2 of the corresponding bounding box of each open door can be calculated. If the open door matches the other vehicle with an open door, the corresponding bounding box of the open door should be completely within the corresponding bounding box of the other vehicle with an open door. Therefore, the value of area S2 divided by area S1 can be calculated as the overlapping area value, and the value of the overlapping area value divided by area S2 can be calculated as the matching detection value. If the matching detection value exceeds the preset threshold, it indicates that the corresponding bounding box of the open door is basically completely within the corresponding bounding box of the other vehicle with an open door, and the two can be determined to be matching vehicles and doors. If the matching detection value does not exceed the preset threshold, it indicates that the corresponding bounding box of the open door is basically not within the corresponding bounding box of the other vehicle with an open door, and the two can be determined to be unmatched vehicles and doors. After performing the above matching operation for all other vehicles with open doors and all open doors, if there is other vehicles with open doors that are not matched with the open doors, then the other vehicles with open doors will be regarded as other vehicles with open doors that were falsely detected; if there is open doors that are not matched with other vehicles with open doors, then the open doors will be regarded as falsely detected open doors.

[0055] It will be understood that the preset threshold value may be any suitable value adjusted according to actual needs, such as 0.95, 0.98, etc., and the scope of the subject matter claimed in the present disclosure is not limited in the above aspects.

[0056] By setting a preset threshold for the matching detection value, a certain amount of redundancy can be retained to avoid mismatches between the open door and other vehicles with open doors due to errors in the detected bounding box, further reducing the false detection rate and improving the accuracy of subsequent warnings to the driver.

[0057] Figure 5 is a flow chart showing a method for determining whether to issue a warning to a vehicle based on one or more bounding boxes and one or more object categories according to an exemplary embodiment. Figure 5 As shown, step S130, based on one or more bounding boxes and one or more object categories, determining whether to warn the vehicle may include, for each open door of other vehicles that match the open door: step S510, determining whether the corresponding bounding box of the open door is located in a preset warning area; and step S520, based on determining that the corresponding bounding box of the open door is located in the preset warning area, warning the vehicle.

[0058] In step S510, the shape and size of the preset warning area can be set according to different attributes of the vehicle and driving parameters such as driving speed and driving acceleration. The larger the size of the preset warning area, the earlier the vehicle can be warned, thereby reducing the occurrence of vehicle-door collision accidents. The smaller the size of the preset warning area, the later the vehicle is warned, thereby avoiding frequent warnings to the vehicle driver. The preset warning area can be set according to actual needs.

[0059] In some examples, the preset warning area may be a trapezoidal warning area, which is based on the principle that objects in an image are larger when they are near and smaller when they are far away; the preset warning area may also be a rectangular warning area, which is based on the consideration of ease of operation.

[0060] In other examples, if the mass of the vehicle is large, the preset warning area may be set to be relatively large, and if the mass of the vehicle is small, the preset warning area may be set to be relatively small.

[0061] In some other examples, the size of the warning area can be adjusted in real time during the vehicle's driving process according to the vehicle's driving speed, driving acceleration, etc. For example, if the vehicle's driving speed is high and / or the driving acceleration is high, the preset warning area can be set to be relatively large, and if the vehicle's driving speed is low and / or the driving acceleration is low, the preset warning area can be set to be relatively small.

[0062] In step S510, it can be determined whether the bounding box is located in the preset warning area based on the position information of the center point of the lower boundary of the bounding box, the geometric center point of the bounding box, and the center point of the upper boundary of the bounding box as described above. For example, if the center point of the lower boundary of the corresponding bounding box of the open door of another vehicle matched with the open door enters the preset warning area, the vehicle is warned.

[0063] According to some embodiments of the present disclosure, the above-mentioned vehicle warning methods 100 and 300 may further include: correcting a preset warning area.

[0064] It will be understood that vehicles such as two-wheeled vehicles and unicycles may experience bumps, tilts, etc. during driving, which may not only affect the results of whether other vehicles, doors, and other objects can be detected, but may also affect the preset warning area, thereby reducing the accuracy of the warning information. For example, when a vehicle turns or drives over a bumpy road, the position of the detected object will change due to the tilt or pitch of the camera. In this case, if it is still determined based on the original preset warning area whether to warn the vehicle, the accuracy of the warning information may be greatly reduced (for example, the corresponding bounding box of the door that is not located in the warning area enters the preset warning area, thereby warning the driver of the vehicle). By correcting the preset warning area, a preset warning area that is more in line with reality at the current moment can be obtained, which is conducive to improving the accuracy of the warning information and avoiding false alarms.

[0065] It will also be understood that when the preset warning area is corrected, the shape and / or size of the preset warning area can be corrected. For example, when the shape of the preset warning area is a trapezoid, the lower boundary of the trapezoidal preset warning area can be moved along the lower boundary of the current image, and the two points of the upper boundary of the trapezoidal preset warning area can be arbitrarily adjusted.

[0066] According to some embodiments of the present disclosure, correcting the preset warning area may include: capturing the position and posture information of the vehicle during driving; and correcting the preset warning area based on the position and posture information.

[0067] In some examples, the IMU attitude sensor can be used to capture the vehicle's posture information during driving. The IMU attitude sensor is a high-performance three-dimensional motion attitude measurement system based on MEMS technology. It contains motion sensors such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass. It can obtain temperature-compensated three-dimensional attitude and orientation data through the embedded ARM processor. Using quaternion-based three-dimensional algorithms and data fusion technology, zero-drift three-dimensional attitude and orientation data represented by quaternions and Euler angles can be output in real time.

[0068] By using the data obtained from the IMU attitude sensor to correct and compensate the preset warning area, the regional displacement deviation caused by the up and down floating and tilting of the body sensor can be eliminated when the vehicle bumps and tilts.

[0069] Figure 6 FIG. 2 shows a schematic diagram of correcting a preset warning area according to an exemplary embodiment. Figure 6 As shown, when the vehicle (not shown) does not bump or tilt, the preset warning area is shown as the area surrounded by the light arrow 610. When the vehicle bumps or tilts, the lower boundary of the preset warning area can be moved along the lower boundary of the current image based on data obtained by the IMU attitude sensor, and the two points of the upper boundary of the preset warning area can be dynamically moved. The corrected preset warning area is shown as the area surrounded by the dark arrow 620.

[0070] According to some embodiments of the present disclosure, the vehicle warning methods 100 and 300 may further include: after determining to issue a warning to the vehicle, displaying one or more objects that trigger the warning.

[0071] Continuing with the above example, if it is determined that the corresponding bounding boxes of one or more open doors are located in a preset warning area and the vehicle needs to be warned, it can be displayed (e.g., highlighted) on a display screen equipped by the vehicle which open doors and corresponding matching other vehicles are likely to cause a collision accident, thereby issuing a warning.

[0072] Furthermore, while displaying one or more objects that trigger the warning, the reminder to the vehicle driver can be strengthened by flashing lights, sounding alarms, etc. For example, after the center point of the lower boundary of the corresponding boundary box of one or more open doors enters the preset warning area, as the distance between the center point and the lower boundary of the preset warning area becomes smaller and smaller, the vehicle can be warned in an increasingly strong manner, for example, the volume and frequency of the warning buzzer can be increased, the flashing frequency of the warning indicator light can be increased, and the highlight prompt can be performed more significantly.

[0073] Figure 7 FIG. 7 shows a structural block diagram of a vehicle warning device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 for issuing an early warning to a vehicle may include: a capturing unit 710, configured to capture one or more images of the vehicle during driving via a sensor; a target detection unit 720, configured to perform target detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and one or more categories of the objects; and an early warning unit 730, configured to determine whether to issue an early warning to the vehicle based on the one or more bounding boxes and the one or more categories of the objects.

[0074] Compared with the traditional ranging sensor, the vehicle warning method according to the embodiment of the present disclosure can capture one or more images during the vehicle driving based on the visual sensor, and locate and identify the object in front of the vehicle and classify the object based on the image visual information. Since the warning based on the target detection, positioning and classification of the image visual information is more efficient, and only a small number of image sensors (for example, one) can be used to achieve the above operations, the system can be simplified, the cost can be reduced, the use is simple and the failure rate can be reduced.

[0075] According to some embodiments of the present disclosure, the target detection unit 720 may include a neural network model unit configured to perform target detection on each of the one or more images based on a neural network model to obtain one or more bounding boxes of one or more objects and one or more categories of the objects.

[0076] According to some embodiments of the present disclosure, the neural network model unit may include: a unit configured to perform target detection on the one or more images based on a student model network, wherein the student model network is obtained by the following operations: training a teacher model network, wherein the teacher model network has a first parameter amount; and obtaining a student model network based on the teacher model network, wherein the student model network has a second parameter amount that is smaller than the first parameter amount.

[0077] According to some embodiments of the present disclosure, the neural network model unit may include: a unit configured to perform model reparameterization on the neural network model to couple multiple sets of parameters corresponding to multiple branches of the neural network model; and a unit configured to perform target detection on each of the one or more images based on the model reparameterized neural network model.

[0078] According to some embodiments of the present disclosure, the category of the one or more objects may include one or more of the following: other vehicles with unopened doors, other vehicles with opened doors, and open doors.

[0079] According to some embodiments of the present disclosure, the device 700 for providing an early warning to a vehicle may further include a matching unit, which is configured to match each other vehicle with an open door and each open door based on determining that the category of one or more objects includes other vehicles with open doors and open doors; and wherein the early warning unit 730 is further configured to determine whether to provide an early warning to the vehicle based on the matching result.

[0080] According to some embodiments of the present disclosure, the matching unit may include a unit configured to perform the following operations for each other vehicle with an open door and each open door: calculate a first area of ​​a corresponding bounding box of the other vehicle with an open door and a second area of ​​a corresponding bounding box of the open door; calculate an overlapping area of ​​the first area and the second area; calculate a quotient of the overlapping area and the second area as a matching detection value, and determine that the other vehicle with an open door matches the open door based on a determination that the matching detection value exceeds a preset threshold; determine that the other vehicle with an open door does not match the open door based on a determination that the matching detection value does not exceed the preset threshold; determine that the other vehicle with an open door is a misdetected vehicle based on a determination that a door that is not opened matches the other vehicle with an open door; and determine that the open door is a misdetected door based on a determination that the other vehicle with an open door matches the open door.

[0081] According to some embodiments of the present disclosure, the early warning unit 730 may include a unit configured to perform the following operations for each open door of other vehicles that match the open door: determining whether the corresponding bounding box of the open door is located in a preset early warning area; and based on determining that the corresponding bounding box of the open door is located in the preset early warning area, issuing an early warning to the vehicle.

[0082] According to some embodiments of the present disclosure, the device 700 for providing early warning to a vehicle may further include a correction unit configured to correct a preset early warning area.

[0083] According to some embodiments of the present disclosure, the correction unit may include a unit configured to perform the following operations: capture the posture information of the vehicle during driving; and correct the preset warning area based on the posture information.

[0084] According to some embodiments of the present disclosure, the device 700 for providing an early warning to a vehicle may further include a unit configured to display one or more objects that trigger the early warning after determining to provide an early warning to the vehicle.

[0085] According to some embodiments of the present disclosure, the vehicle may be a two-wheeled vehicle or a unicycle.

[0086] It should be understood that Figure 7 The various units of the device 700 for providing early warning to a vehicle shown in FIG. 7 can be used in conjunction with the reference Figure 1 and Figure 3 The steps in the methods 100 and 300 for early warning of a vehicle are described to correspond to each other. Therefore, the operations, features and advantages described above for the methods 100 and 300 are also applicable to the device 700 for early warning of a vehicle and the units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.

[0087] Yet another aspect of the present disclosure may include an electronic device, which may include a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the above-mentioned method for providing early warning to a vehicle.

[0088] Yet another aspect of the present disclosure may include a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the above-mentioned method for providing early warning to a vehicle are implemented.

[0089] Yet another aspect of the present disclosure may include a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned method for providing early warning to a vehicle are implemented.

[0090] Reference Figure 8 , a computing device 800 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The computing device 800 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, an in-vehicle computer, an access control system, an attendance device, or any combination thereof. The above-mentioned device for warning a vehicle can be implemented in whole or in part by the computing device 800 or a similar device or system. Although the computing device 800 represents an example of several types of computing platforms, the computing device 800 can include more than Figure 8 Greater or fewer elements and / or different arrangements of elements may be shown, and the scope of the claimed subject matter is not limited in these respects.

[0091] In some embodiments, the computing device 800 may include an element connected to or in communication with the bus 802 (possibly via one or more interfaces). For example, the computing device 800 may include a bus 802, one or more processors 804, one or more input devices 806, and one or more output devices 808. The one or more processors 804 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 806 may be any type of device that can input information to the computing device 800, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote control. The output device 808 may be any type of device that can present information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 800 may also include or be connected to a non-transitory storage device 810. The non-transitory storage device may be any storage device that is non-transitory and can implement data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape, or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 810 may be detachable from the interface. The non-transitory storage device 810 embodies one or more non-transitory computer-readable media on which a program including instructions is stored, which, when executed by one or more processors of the computing device 800, causes the computing device 800 to perform the above-mentioned method 100, 300 for warning a vehicle and its variants. The computing device 800 may also include a communication device 812. The communication device 812 can be any type of device or system that enables communication with an external device and / or with a network, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0092] In some embodiments, computing device 800 may also include working memory 814, which may be any type of memory that can store programs (including instructions) and / or data useful for operation of processor 804, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0093] Software elements (programs) may be located in the working memory 814, including but not limited to an operating system 816, one or more applications 818, drivers and / or other data and codes. Instructions for executing the above methods and steps may be included in one or more applications 818, and the electronic circuit of the above-mentioned device for warning the vehicle may be implemented by reading and executing the instructions of one or more applications 818 by the processor 804. The executable code or source code of the instructions of the software element (program) may be stored in a non-transitory computer-readable storage medium (such as the above-mentioned storage device 810), and may be stored in the working memory 814 when executed (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) may also be downloaded from a remote location.

[0094] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0095] It should also be understood that the aforementioned method can be implemented in a server-client mode. For example, the client can use a camera to collect image data and send the image data to the server for subsequent processing. The client can also perform a portion of the processing in the aforementioned method and send the processed data to the server. The server can receive data from the client and execute the aforementioned method or another portion of the aforementioned method, and return the execution result to the client. The client can receive the execution result of the method from the server and present it to the user, for example, through an output device.

[0096] It should also be understood that the components of computing device 800 can be distributed over a network. For example, some processing can be performed using one processor, while other processing can be performed by another processor remote from the one processor. Other components of computing device 800 can also be similarly distributed. In this way, computing device 800 can be understood as a distributed computing system that performs processing on multiple processors at multiple locations.

[0097] Some exemplary aspects of the disclosure are described below.

[0098] Aspect 1. A method for early warning of a vehicle, comprising:

[0099] capturing, via a sensor, one or more images of the vehicle during travel;

[0100] performing object detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and categories of the one or more objects; and

[0101] Based on the one or more bounding boxes and the categories of the one or more objects, it is determined whether to issue an early warning to the vehicle.

[0102] Aspect 2. The method according to aspect 1, wherein performing target detection on each of the one or more images to obtain the one or more bounding boxes of the one or more objects and the categories of the one or more objects comprises:

[0103] Perform object detection on each of the one or more images based on a neural network model to obtain the one or more bounding boxes of the one or more objects and the categories of the one or more objects.

[0104] Aspect 3. The method according to aspect 2, wherein performing target detection on each of the one or more images based on the neural network model comprises:

[0105] Performing object detection on the one or more images based on a student model network, wherein the student model network is obtained by the following operations:

[0106] training a teacher model network, the teacher model network having a first number of parameters; and

[0107] A student model network is obtained based on the teacher model network, wherein the student model network has a second parameter amount that is smaller than the first parameter amount.

[0108] Aspect 4. The method according to aspect 2, wherein performing target detection on each of the one or more images based on the neural network model comprises:

[0109] Reparameterizing the neural network model to couple multiple groups of parameters corresponding to multiple branches of the neural network model; and

[0110] Based on the neural network model with model reparameterization, target detection is performed on each of the one or more images.

[0111] Aspect 5. The method according to aspect 1, wherein the categories of the one or more objects include one or more of the following: other vehicles with unopened doors, other vehicles with opened doors, and open doors.

[0112] Aspect 6. The method according to aspect 5 further includes:

[0113] Based on determining that the category of the one or more objects includes other vehicles with open doors and open doors, matching each other vehicle with open doors and each open door, and

[0114] Wherein, determining whether to issue an early warning to the vehicle is further based on the matching result.

[0115] Aspect 7. The method according to aspect 6, wherein, based on determining that the categories of the one or more objects include other vehicles with open doors and open doors, matching each other vehicle with open doors and each open door comprises:

[0116] For each of the other vehicles with open doors and each of the open doors:

[0117] Calculating a first area of ​​a corresponding bounding box of the other vehicle with the open door and a second area of ​​a corresponding bounding box of the open door;

[0118] Calculating an overlapping area between the first area and the second area;

[0119] calculating the quotient of the overlapping area and the second area as a matching detection value,

[0120] Determining that the other vehicle with the open door matches the open door based on determining that the matching detection value exceeds a preset threshold;

[0121] Determining that the matching detection value does not exceed the preset threshold value, determining that the other vehicle with the opened door does not match the opened door;

[0122] According to determining that the vehicle door that is not opened matches the other vehicle with the opened door, determining that the other vehicle with the opened door is a misdetected vehicle; and

[0123] According to determining that other vehicles without opened doors match the opened door, the opened door is determined to be a misdetected door.

[0124] Aspect 8. The method according to aspect 6, wherein, based on the one or more bounding boxes and the categories of the one or more objects, determining whether to warn the vehicle comprises, for each open door of other vehicles matched to the open door:

[0125] Determining whether a corresponding bounding box of the opened door is located in a preset warning area; and

[0126] According to determining that the corresponding boundary box of the opened door is located in the preset warning area, the vehicle is warned.

[0127] Aspect 9. The method according to aspect 8 further includes:

[0128] The preset warning area is corrected.

[0129] Aspect 10. The method according to aspect 9, wherein correcting the preset warning area comprises:

[0130] Capturing the position information of the vehicle during driving; and

[0131] The preset warning area is corrected based on the posture information.

[0132] Aspect 11. The method according to any one of aspects 5 to 10, further comprising:

[0133] After determining to issue an early warning to the vehicle, the one or more objects triggering the early warning are displayed.

[0134] Aspect 12. The method according to any one of aspects 1 to 10, wherein the vehicle is a two-wheeled vehicle or a unicycle.

[0135] Aspect 13. A device for providing early warning to a vehicle, comprising:

[0136] a capturing unit configured to capture one or more images during driving of the vehicle via a sensor;

[0137] an object detection unit configured to perform object detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and categories of the one or more objects; and

[0138] The warning unit is configured to determine whether to issue a warning to the vehicle based on the one or more bounding boxes and the categories of the one or more objects.

[0139] Aspect 14. An electronic device comprising:

[0140] a memory, a processor and a computer program stored on the memory,

[0141] The processor is configured to execute the computer program to implement the steps of the method according to any one of aspects 1 to 12.

[0142] Aspect 15. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 12 are implemented.

[0143] Aspect 16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 12 are implemented.

[0144] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for early warning of a vehicle, comprising: capturing, via a sensor, one or more images of the vehicle during travel; performing object detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and categories of the one or more objects; as well as Based on the one or more bounding boxes and the categories of the one or more objects, it is determined whether to issue an early warning to the vehicle.

2. The method according to claim 1, wherein: Performing object detection on each of the one or more images to obtain the one or more bounding boxes of the one or more objects and the categories of the one or more objects includes: Perform object detection on each of the one or more images based on a neural network model to obtain the one or more bounding boxes of the one or more objects and the categories of the one or more objects.

3. The method according to claim 2, wherein: Performing object detection on each of the one or more images based on the neural network model includes: Performing object detection on the one or more images based on a student model network, wherein the student model network is obtained by the following operations: training a teacher model network, the teacher model network having a first number of parameters; and A student model network is obtained based on the teacher model network, wherein the student model network has a second parameter amount that is smaller than the first parameter amount.

4. The method according to claim 2, wherein: Performing object detection on each of the one or more images based on the neural network model includes: Reparameterizing the neural network model to couple multiple groups of parameters corresponding to multiple branches of the neural network model; and Based on the neural network model with model reparameterization, target detection is performed on each of the one or more images.

5. The method according to claim 1, wherein: The categories of the one or more objects include one or more of the following: other vehicles with unopened doors, other vehicles with opened doors, and open doors.

6. The method according to claim 5, further comprising: Based on determining that the category of the one or more objects includes other vehicles with open doors and open doors, matching each other vehicle with open doors and each open door, and Wherein, determining whether to issue an early warning to the vehicle is further based on the matching result.

7. A device for early warning of a vehicle, comprising: a capturing unit configured to capture one or more images during driving of the vehicle via a sensor; an object detection unit configured to perform object detection on each of the one or more images to obtain one or more bounding boxes of one or more objects and categories of the one or more objects; as well as The warning unit is configured to determine whether to issue a warning to the vehicle based on the one or more bounding boxes and the categories of the one or more objects.

8. An electronic device comprising: a memory, a processor and a computer program stored on the memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images