Vehicle early warning method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a vehicle early warning method and device, electronic equipment, a computer readable storage medium and a computer program product. The vehicle early warning method comprises the following steps: capturing a plurality of images during the driving period of a vehicle through a sensor, wherein the sensor is positioned at the tail part of the vehicle; performing target detection on each image in the plurality of images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes; based on the one or more bounding boxes and the position information thereof, determining the motion trails of the one or more objects; and determining whether to carry out early warning on the vehicle based on the motion trails of the one or more objects.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a vehicle early warning method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] When a vehicle is driving on the road, the driver may not be able to observe the moving objects behind him in time, such as vehicles and pedestrians, due to reasons such as line of sight obstruction. In this case, the driver may change lanes and collide with the moving object behind him.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of the present disclosure, a vehicle warning method is provided, comprising: capturing multiple images of the vehicle during driving via a sensor, the sensor being located at the rear of the vehicle; performing target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes; determining a motion trajectory of the one or more objects based on the one or more bounding boxes and their position information; and determining whether to issue a warning to the vehicle based on the motion trajectory of the one or more objects.

[0005] According to another aspect of the present disclosure, a vehicle warning device is provided, including: a capture unit, configured to capture multiple images of the vehicle during driving via a sensor, the sensor being located at the rear of the vehicle; a target detection unit, configured to perform target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes; a motion trajectory determination unit, configured to determine the motion trajectory of the one or more objects based on the one or more bounding boxes and their position information; and a warning unit, configured to determine whether to warn the vehicle based on the motion trajectory of the one or more objects.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the above-mentioned vehicle warning method.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the above-mentioned vehicle warning method when executed by a processor.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the above-mentioned vehicle warning method when executed by a processor.

[0009] Further features and advantages of the present disclosure will become apparent from the exemplary embodiments described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0011] Figure 1 is a flow chart showing a vehicle warning method according to an exemplary embodiment;

[0012] Figure 2 is a schematic diagram illustrating model distillation according to an exemplary embodiment;

[0013] Figure 3 is a flow chart illustrating determining a motion trajectory of one or more objects based on one or more bounding boxes and their position information according to an exemplary embodiment;

[0014] Figure 4 is a flow chart showing a vehicle warning method according to another exemplary embodiment;

[0015] Figure 5 is a flowchart showing whether to warn a vehicle based on the motion trajectory of one or more objects according to an exemplary embodiment;

[0016] Figure 6 is a flowchart showing a method of determining whether to issue a warning to a vehicle based on a motion trajectory of one or more objects according to another exemplary embodiment;

[0017] Figure 7 A structural block diagram of a vehicle warning device according to an embodiment of the present disclosure is shown;

[0018] Figure 8 is a block diagram showing a structure of a computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0021] When a vehicle is driving on the road, the driver may not be able to observe the moving objects behind him in time, such as vehicles and pedestrians, due to reasons such as blocked vision. In this case, the driver may change lanes and collide with the moving object behind him. In response to this situation, blind spot warning systems are beginning to be installed on vehicles to provide safety warnings to drivers by detecting moving objects behind the vehicle in real time.

[0022] Existing vehicle blind spot warning systems are mainly based on ranging sensors such as radar or ultrasonic waves to detect the distance of the moving object behind the vehicle, and issue a safety warning to the driver based on the distance and speed information. However, the inventors found that this type of ranging sensor cannot perform category recognition and contour perception for the moving object behind the vehicle, so that it is impossible to issue an accurate and detailed warning to the driver. On the other hand, some vehicles (such as two-wheeled vehicles such as motorcycles, electric bicycles, two-wheeled balancing cars, electric scooters, and unicycles such as single-wheeled balancing cars) usually have a tilt phenomenon during driving (such as turning or driving over bumpy roads), which may cause the accuracy of this type of ranging sensor to decrease, and then produce a large error in moving object detection. If multiple ranging sensors are set on the vehicle in order to improve detection and warning accuracy, the cost will be increased and it will be difficult to align and fuse multiple groups of inputs, increasing the probability of failure of the warning system.

[0023] In order to solve the above technical problems, the present disclosure provides a new vehicle warning method and device, electronic device, computer-readable storage medium and computer program product. It will be clearly seen in the detailed description below that the vehicle warning method according to the embodiment of the present disclosure can capture multiple images during vehicle driving, locate and identify objects behind the vehicle and judge their motion trends based on such visual information, and provide safety warnings to the driver based on the motion trends of one or more objects. Compared with traditional ranging sensors, target detection and positioning based on image-based visual information and judgment of motion trends will be more efficient. At the same time, since the vehicle according to the embodiment of the present disclosure only uses one image sensor as a target detection sensor, it can reduce costs, be easy to use and have a low failure rate.

[0024] An exemplary embodiment of the vehicle warning method disclosed herein will be further described below with reference to the accompanying drawings.

[0025] Figure 1 FIG. 1 is a flow chart of a vehicle warning method 100 according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the vehicle warning method 100 may include: step S110, capturing multiple images of the vehicle during driving via a sensor, wherein the sensor is located at the rear of the vehicle; step S120, performing target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes; step S130, determining the motion trajectory of one or more objects based on the one or more bounding boxes and their position information; and step S140, determining whether to warn the vehicle based on the motion trajectory of the one or more objects.

[0026] Compared with traditional ranging sensors, the vehicle warning method according to the embodiment of the present disclosure can capture multiple images during vehicle driving based on visual sensors, and based on this, locate and identify objects behind the vehicle and judge the movement trend. Since target detection and positioning based on image-based visual information and judgment of movement trends are more efficient, and only one image sensor is used to achieve the above operations, it can reduce costs, simplify operations and reduce failure rates.

[0027] According to some embodiments of the present disclosure, the vehicle may be a two-wheeled vehicle or a unicycle.

[0028] In step S110, the sensor may be any suitable sensor capable of capturing an image, such as a camera, a video camera, a webcam, etc. Such sensors may be independent devices as described above, or may be included in various electronic devices (e.g., mobile phones, computers, personal digital assistants, tablet computers, wearable devices, etc.). The scope of the subject matter claimed in the present disclosure is not limited in the above aspects.

[0029] In step S110, the sensor may be located at any position of the rear of the vehicle as long as it can capture the moving object behind the vehicle in real time, such as being installed above the rear wheels of the vehicle, or installed on the rear storage box of the vehicle, etc., preferably installed in the direction of the central axis of the vehicle.

[0030] According to some embodiments of the present disclosure, in step S120, performing target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and location information of the one or more bounding boxes may include: performing target detection on each of the multiple images based on a neural network model to obtain one or more bounding boxes of one or more objects and location information of the one or more bounding boxes.

[0031] In some examples, the position information of the bounding box of each object may be the center point of the bottom border of the bounding box. In other examples, the position information of the bounding box of each object may be the geometric center point of the bounding box. In still other examples, the position information of the bounding box of each object may be the center point of the top border of the bounding box. The position information may be, for example, the two-dimensional coordinates or three-dimensional coordinates of the center point.

[0032] It will be understood that the position information of the bounding box may refer to coordinate information of any suitable points of the bounding box, and the scope of the subject matter claimed in the present disclosure is not limited in the foregoing respects.

[0033] It will also be understood that the neural network model can be a YOLOv8 model or other models based on deep neural network algorithms to perform target detection on each image.

[0034] According to some embodiments of the present disclosure, performing target detection on each of a plurality of images based on a neural network model may include: performing target detection on a plurality of images based on a student model network, wherein the student model network is obtained by the following operations: training a teacher model network, wherein the teacher model network has a first parameter amount; and obtaining a student model network based on the teacher model network, wherein the student model network has a second parameter amount which is smaller than the first parameter amount.

[0035] Figure 2 is a schematic diagram showing model distillation according to an exemplary embodiment. Figure 2 The target detection model 200 may include a teacher model network 210 with relatively more parameters and stronger learning ability and a student model network 220 with relatively fewer parameters and weaker learning ability. In the training phase, the teacher model network 210 is first trained using training samples, and then the error is distilled based on the following formula to obtain the student model network 220 through "knowledge transfer":

[0036]

[0037] in, is the cross entropy, p(x) is the output of the student model network, q(x) is the output of the teacher model network, l(x) is the logits output of the student model network, and z(x) is the logits output of the teacher model network.

[0038] Based on the teacher model network to guide the training of the student model network, the feature representation "knowledge distillation" learned by the complex and strong learning teacher model network can be further passed to the student model network with a small number of parameters and weak learning ability, thereby obtaining a fast and expressive student model network. Accordingly, compared with the output results obtained based only on the student model network, the accuracy of the model can be improved (for example, by 10%).

[0039] According to some embodiments of the present disclosure, performing target detection on each of a plurality of images based on a neural network model may include: performing quantization processing on the neural network model to convert an input data type of the neural network model into a predefined data type.

[0040] Typically, the input data type to a neural network model can be a floating-point data type, which may be more complex to calculate in the application model and has limited application scenarios. By quantizing such models, that is, converting floating-point data types into data types (e.g., int 8) with low bit widths (e.g., 8 bits or 4 bits) suitable for edge devices, the complexity of operations can be greatly reduced during calculations such as convolution, thereby reducing the size of the model, enabling deployment on hardware-constrained devices and reducing computing time.

[0041] For example, the data type of the parameter input to the neural network model can be fp 32, and then the data type of the parameter can be converted to int 8 through quantization operation, and then the quantized data is convolved (the result data type is still int 8), and finally the data type of the convolution result is converted back to fp 32 for output through dequantization. In this example, since low bit width type data is used instead of floating point type data for calculation in the neural network, the complexity and size of the model can be reduced while maintaining the model performance.

[0042] Furthermore, after the model is quantized, it can be further fine-tuned to avoid the loss of accuracy due to quantization loss.

[0043] According to some embodiments of the present disclosure, the vehicle warning method 100 may further include: in step S120, performing target detection on each of the multiple images to obtain categories of one or more objects.

[0044] In some examples, a neural network model can be used to detect objects in images. When training such a neural network, objects such as pedestrians, cars, and two-wheeled vehicles can be identified based on a large number of data sets, but other objects are not. This is because in actual application scenarios, objects associated with vehicle driving safety are usually more concerned, which can reduce the size of the training data set and improve the accuracy of target recognition.

[0045] In some examples, after performing target detection on an image, a category ID may be assigned to each identified object, such as pedestrian 1, pedestrian 2, car 1, car 2, etc., to facilitate subsequent target tracking of these objects.

[0046] Figure 3 is a flow chart showing a method for determining a motion trajectory of one or more objects based on one or more bounding boxes and their position information according to an exemplary embodiment. Figure 3 As shown, step S130, based on one or more bounding boxes and their position information, determining the motion trajectory of one or more objects may include, for each object in the one or more objects in each image: step S310, determining the confidence between the object and each object in the image captured before the image; step S320, determining that the object matches the corresponding object in the image captured before the image based on determining that the confidence is greater than a first confidence threshold; step S330, determining whether the confidence is greater than a second confidence threshold based on determining that the confidence is not greater than the first confidence threshold, wherein the second confidence threshold is less than the first confidence threshold; step S340, determining that the object matches the corresponding object in the image captured before the image based on determining that the confidence is greater than the second confidence threshold; and step S350, determining the motion trajectory of the object based on the bounding boxes of the matched object in multiple images and their position information.

[0047] In step S310, the confidence may be a value between 0 and 1, and indicates the degree of certainty of the model about the detected target. For example, the confidence value may be calculated by the following two factors: (1) whether there is an object in the bounding box obtained by target detection. If the bounding box accurately surrounds an object, the confidence value may be high, while if there is no object or only a part of the object in the bounding box, the confidence value may be low; (2) the matching degree of the object in the bounding box with each object in the previously captured image. If the matching degree of the object in the bounding box with an object in the previously captured image is high, the confidence value is high, while if the matching degree of the object in the bounding box with an object in the previously captured image is low, the confidence value is low.

[0048] In step S320, the first confidence value threshold may be any suitable value, such as 0.3, 0.35, etc. If the confidence between the object in the identified bounding box and an object in an image captured before the image is greater than the confidence threshold, it indicates that the two objects match.

[0049] In actual application scenarios, an object in an image captured at a certain moment may be occluded. In this case, the confidence between the occluded object and any object in the previously captured image may be less than the preset confidence threshold, but this does not mean that the object is not recognized in the frame image. In order to avoid the loss of an object in a frame image due to occlusion, a second confidence threshold that is less than the first confidence threshold may be set, as described in step S330 below.

[0050] In steps S330-S340, the second confidence threshold may be any suitable value less than the first confidence threshold, or may be a threshold interval, for example, the second confidence threshold may be a threshold interval ranging from 0.1 to 0.3. When the confidence between the object in the identified bounding box and any object in the image captured before the image is less than the first confidence threshold, the confidence value may be compared with the second confidence threshold. If the confidence value is not greater than the second confidence threshold, it indicates that there is no object matching the object in the previously captured image, that is, the possibility of the object being lost in the frame image is relatively high. If the confidence value is greater than the second confidence value, it indicates that there is an object matching the object in the previously captured image, but the confidence value is low due to occlusion or other reasons.

[0051] In step S350, after performing confidence calculation on each object in each image and determining the matching result, the motion trajectory of each object can be determined based on the position information of the bounding box of the matched object in multiple images. As described above, the position information of the bounding box can refer to the coordinates of the center point of the lower boundary of the bounding box. By connecting the center points of the lower boundaries of the bounding boxes of a matched object in multiple images, the motion trajectory of the object can be obtained.

[0052] It will be understood that the "image captured before this image" in the present disclosure may refer to an image captured in the previous frame, or may refer to the previous multiple frames of images captured continuously, and the scope of the subject matter claimed in the present disclosure is not limited in the above aspects.

[0053] In the target tracking process, by performing two confidence comparisons on the detected objects, the loss of objects in a certain frame image due to occlusion can be avoided, thereby improving the accuracy of target tracking. At the same time, since only the bounding box of the detected object and its position information are used in the target tracking process (preferably, the target can also be tracked in combination with the category of the object), without other complex operations, the efficiency of target tracking can be greatly improved.

[0054] According to some embodiments of the present disclosure, the vehicle warning method 100 may further include: acquiring motion information of a sensor; and wherein, step S130, determining the motion trajectory of one or more objects based on one or more bounding boxes and their position information may further include: for each of the one or more objects in each image, determining that the object matches a corresponding object in an image captured before the image based on both confidence and motion information.

[0055] In some examples, the target can be tracked by further combining the motion information of sensors such as cameras. For example, vehicles such as two-wheeled vehicles may experience bumps, tilts, etc. during driving. By considering the motion information of the sensor to compensate for the target tracking result, the accuracy of target tracking can be further improved.

[0056] Figure 4 FIG. 1 is a flow chart showing a vehicle warning method according to another exemplary embodiment. Figure 4 As shown, the vehicle warning method 400 may include the reference Figure 1 Steps S410 - S440 are similar to steps S110 - S140 in the described vehicle warning method 100 ; and step S450 , correcting the motion trajectory of one or more objects.

[0057] As mentioned above, the vehicle may experience bumps, tilts, etc. during driving, which may not only affect whether the object can be tracked, but also affect the accuracy of the position of the tracked object. For example, when the vehicle turns or drives over a bumpy road, the trajectory of the detected object will fluctuate abnormally due to the tilt or pitch of the camera. By correcting the trajectory of one or more objects after determining their motion trajectory and compensating for the abnormally fluctuating trajectory, the motion trajectory of the object can be made smoother and more accurate, which is conducive to improving the accuracy of the warning information.

[0058] According to some embodiments of the present disclosure, step S450, correcting the motion trajectory of one or more objects may include: capturing the posture information of the vehicle during driving; and correcting the motion trajectory of the one or more objects based on the posture information.

[0059] In some examples, the IMU attitude sensor can be used to capture the vehicle's posture information during driving. The IMU attitude sensor is a high-performance three-dimensional motion attitude measurement system based on MEMS technology. It contains motion sensors such as a three-axis gyroscope, a three-axis accelerometer, and a three-axis electronic compass. It can obtain temperature-compensated three-dimensional attitude and orientation data through the embedded ARM processor. Using quaternion-based three-dimensional algorithms and data fusion technology, zero-drift three-dimensional attitude and orientation data represented by quaternions and Euler angles can be output in real time.

[0060] By using the data obtained by the IMU attitude sensor to correct and compensate the trajectory of the tracked object, the impact of the up and down floating and tilting of the body sensor can be eliminated when the vehicle bumps and tilts.

[0061] According to some other embodiments of the present disclosure, step S450, correcting the motion trajectory of one or more objects may include: for each object of the one or more objects, based on determining that the bounding box of the object is not detected in at least one image of the multiple images, compensating for the position information of the bounding box of the object in at least one image based on the position information of the bounding box of the object detected in the first image and the position information of the bounding box of the object detected in the second image to obtain a corrected motion trajectory of the object, wherein the first image is an image captured before at least one image of the multiple images, and the second image is an image captured after at least one image of the multiple images.

[0062] As mentioned above, when a vehicle is driving, an object in an image captured at a certain moment may be blocked. If the object is completely blocked in the corresponding image (i.e., the corresponding object is matched in both the previously captured and the subsequently captured images), the motion trajectory of the object will be lost.

[0063] According to an embodiment of the present disclosure, the lost trajectory points of the object in the image may be interpolated and supplemented based on the position information of the bounding box of the corresponding object in the previously captured image and the position information of the bounding box of the corresponding object in the subsequently captured image.

[0064] In some examples, a linear interpolation algorithm may be used to interpolate and supplement the missing trajectory points of the object in the image. For example, if the position coordinates of the lower boundary of the bounding box of the corresponding object in the previously captured adjacent image are (x1, y1), and the position coordinates of the lower boundary of the bounding box of the corresponding object in the subsequently captured adjacent image are (x2, y2), the coordinate information of the missing trajectory points in the intermediate image may be obtained by calculating the slope m = (y2-y1) / (x2-x1). Continuing with this example, considering that the time interval between capturing adjacent image frames is short, x = (x1+x2) / 2 may be used as the horizontal coordinate of the missing trajectory point of the intermediate image, and the formula y = y1+m*(x-x1) may be applied to calculate the estimated vertical coordinate of the missing trajectory point.

[0065] In other examples, other nonlinear interpolation algorithms may also be used to interpolate and supplement the missing trajectory points of the object in the image. For example, nonlinear fitting may be performed based on the position coordinates of the lower boundary of the bounding box of the corresponding object in the adjacent multiple images previously captured and the position coordinates of the lower boundary of the bounding box of the corresponding object in the adjacent multiple images subsequently captured. Similarly, x=(x1+x2) / 2 is used as the horizontal coordinate of the missing trajectory point of the intermediate image, and the nonlinear fitting formula is applied to calculate the estimated vertical coordinate of the missing trajectory point.

[0066] It will be understood that any one or more previously captured images and any one or more subsequently captured images may be selected according to actual needs to interpolate and supplement the missing track points of a certain object in a certain image. For example, if the operation is expected to be simple, a linear interpolation algorithm may be selected; if the accuracy of the supplemented track points is expected to be further improved, multiple previously captured images and multiple subsequently captured images may be selected.

[0067] Through the above-mentioned trajectory correction and trajectory missing point supplementation, the trajectory of the object can be made smoother and more continuous, so as to be better used for the motion trend prediction of the object and improve the accuracy of the early warning information.

[0068] Figure 5 FIG. 1 is a flow chart showing a method for determining whether to issue a warning to a vehicle based on the motion trajectory of one or more objects according to an exemplary embodiment. Figure 5As shown, step S140, determining whether to warn the vehicle based on the motion trajectory of one or more objects may include: step S510, determining, for each of the one or more objects, whether the bounding box of the object is located at the boundary of the corresponding image; step S520, determining whether to warn the vehicle based on a width change of the bounding box of the object in the corresponding image based on a determination that the bounding box of the object is located at the boundary of the corresponding image; and step S530, determining whether to warn the vehicle based on the motion trajectory of the bounding box of the object based on a determination that the bounding box of the object is not located at the boundary of the corresponding image.

[0069] According to some embodiments of the present disclosure, the position information of one or more bounding boxes may be the midpoint of the lower boundary of the corresponding bounding box, and step S510, for each of the one or more objects, determining whether the bounding box of the object is located at the boundary of the corresponding image may include: determining that the bounding box of the object is located at the boundary of the corresponding image based on determining that at least a portion of the bounding box of the object is located outside the left boundary or the right boundary of the corresponding image; and determining that the bounding box of the object is not located at the boundary of the corresponding image based on determining that the entire bounding box of the object is located in the corresponding image.

[0070] Specifically, when performing object detection on an image, one or more objects may be located outside the sensor's field of view. Accordingly, the bounding boxes of the one or more objects obtained will be located at the left or right boundary of the corresponding image, that is, there is an object truncation phenomenon. In contrast, if one or more objects are all located within the sensor's field of view, the bounding boxes of the one or more objects obtained will all be located inside the corresponding image.

[0071] By setting different warning algorithms according to the different positions of the object in the image, a more accurate warning result can be obtained. The following will describe in detail how to predict the movement trend of the object and how to warn the vehicle according to whether the object is located at the boundary of the corresponding image.

[0072] According to some embodiments of the present disclosure, step S520, determining whether to issue an alert to the vehicle based on a change in the width of the object's bounding box in the corresponding image according to determining that the bounding box of the object is located at the boundary of the corresponding image, may include: determining whether the width of the object's bounding box in the corresponding image is less than a preset width threshold; issuing an alert to the vehicle according to determining that the width of the object's bounding box in the corresponding image is less than the preset width threshold; and determining whether the object's motion trajectory is away from the vehicle or close to the vehicle according to determining that the width of the object's bounding box in the corresponding image is not less than the preset width threshold; not issuing an alert to the vehicle according to determining that the object's motion trajectory is away from the vehicle; and issuing an alert to the vehicle when the width of the object's bounding box in the corresponding image is less than the preset width threshold according to determining that the object's motion trajectory is close to the vehicle.

[0073] In some examples, the preset width threshold may be any suitable pixel value set according to requirements, such as 5, 8, 10, etc. The larger the preset width threshold, the earlier the vehicle can be warned, thereby reducing the occurrence of vehicle collision accidents. The smaller the preset width threshold, the later the vehicle is warned, thereby avoiding frequent warnings to the vehicle driver.

[0074] Continuing with the above example, when the preset width threshold is 5 and the bounding box of an object is located at the boundary of the corresponding image (for example, the left boundary), if the width of the bounding box of the object in the corresponding image is less than 5 pixels, it can be determined that the object is already in the warning area of ​​the vehicle, and the driver of the vehicle can be warned. If the width of the bounding box of the object in the corresponding image is not less than 5 pixels, it can be determined that the distance between the object and the vehicle is relatively far. At this time, the movement trend of the object can be judged, that is, whether to warn the driver of the vehicle can be determined by judging whether the object is close to the vehicle or far away from the vehicle.

[0075] According to some embodiments of the present disclosure, step S520, determining whether to issue an early warning to the vehicle based on a change in the width of the object's bounding box in the corresponding image according to determining that the bounding box of the object is located at the boundary of the corresponding image may include: determining whether the width of the object's bounding box in the corresponding image is less than a preset width threshold; determining whether a change in the width of the object's bounding box in the corresponding image and a previous adjacent image of the corresponding image exceeds a preset width change threshold according to determining that the width of the object's bounding box in the corresponding image is less than the preset width threshold; determining whether the movement trajectory of the object is moving away from the vehicle or approaching the vehicle according to determining that the movement trajectory of the object is moving away from the vehicle, not issuing an early warning to the vehicle; and issuing an early warning to the vehicle according to determining that the movement trajectory of the object is approaching the vehicle; and not issuing an early warning to the vehicle according to determining that the width change of the object's bounding box in the corresponding image and a previous adjacent image of the corresponding image does not exceed the preset width change threshold.

[0076] In some examples, when the bounding box of an object is located at the boundary of a corresponding image, if the width of the bounding box of the object in the corresponding image is not less than a preset width threshold, it is possible to determine whether to issue an early warning to the vehicle in combination with the movement speed of the object. The movement speed of the object can be determined by calculating the width change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image. In the case where the movement speed of the object is relatively large, it is possible to further determine whether the movement trajectory of the object is away from the vehicle or close to the vehicle. If it is away from the vehicle, no early warning may be issued to the vehicle, and if it is close to the vehicle, an early warning may be issued to the vehicle. In the case where the movement speed of the object is relatively small, it can be considered that the movement speed is caused by vehicle body shaking or bumping, etc., and will not cause a large collision risk to the vehicle, so no early warning may be issued to the vehicle.

[0077] In some examples, the preset width change threshold may also be any suitable pixel value set according to requirements. The smaller the preset width change threshold, the earlier the vehicle warning is issued, thereby reducing the occurrence of vehicle collision accidents. The larger the preset width threshold, the later the vehicle warning is issued, thereby avoiding frequent warnings to the vehicle driver.

[0078] When the vehicle body shakes or bumps, it usually causes the bounding box of the object in the corresponding image to change (for example, the width changes). However, this change usually does not affect the collision risk factor. By setting a preset width change threshold, early and frequent warnings to the vehicle driver in this case can be avoided.

[0079] It will be appreciated that the determination of whether to issue an early warning to the vehicle may also be made in conjunction with the object's motion acceleration. For example, the object's motion acceleration may be further determined, and the vehicle may be issued an early warning when the object's motion acceleration is large, while the vehicle may not be issued an early warning when the object's deceleration is large (i.e., the object is decelerating).

[0080] According to some embodiments of the present disclosure, based on determining that the width of the bounding box of the object in the corresponding image becomes larger, it can be determined that the movement trajectory of the object is moving away from the vehicle, and wherein, based on determining that the width of the bounding box of the object in the corresponding image becomes smaller, it can be determined that the movement trajectory of the object is moving toward the vehicle.

[0081] According to some embodiments of the present disclosure, step S530, determining whether to issue an early warning to the vehicle based on the motion trajectory of the object's bounding box according to determining that the bounding box of the object is not located at the boundary of the corresponding image, may include: determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the position information of the object's bounding box; issuing an early warning to the vehicle according to determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold; and determining whether the motion trajectory of the object is away from the vehicle or close to the vehicle according to the distance between the object and the vehicle is not less than the preset trajectory motion threshold; not issuing an early warning to the vehicle according to determining that the motion trajectory of the object is away from the vehicle; and issuing an early warning to the vehicle according to determining that the motion trajectory of the object is close to the vehicle when the distance exceeds the preset trajectory motion threshold.

[0082] Similarly, the preset trajectory motion threshold value can also be any suitable pixel value set according to the requirements, such as 5, 8, 10, etc. The larger the preset trajectory motion threshold value is, the earlier the vehicle can be warned, thereby reducing the occurrence of vehicle collision accidents. The smaller the preset trajectory motion threshold value is, the later the vehicle is warned, thereby avoiding frequent warnings to the vehicle driver.

[0083] Continuing with the above example, when the preset trajectory motion threshold is 5 and the bounding box of an object is not located at the boundary of the corresponding image, if the distance between the object and the vehicle is determined to be less than 5 pixels based on the position information (e.g., coordinate information) of the bounding box of the object, it can be determined that the object is already in the warning area of ​​the vehicle, and the driver of the vehicle can be warned. If the distance between the object and the vehicle is determined to be not less than 5 pixels based on the position information (e.g., coordinate information) of the bounding box of the object, the motion trend of the object can be further judged, and whether to warn the driver of the vehicle can be determined by judging whether the object is close to the vehicle or away from the vehicle.

[0084] It will be understood that the preset width threshold and the preset trajectory motion threshold can be the same value or different values. By using an adaptive threshold to provide warnings based on the area where the object is located, it is possible to better adapt to scene requirements and more accurately provide warnings to vehicle drivers, while avoiding unnecessary warning times as much as possible.

[0085] According to some embodiments of the present disclosure, step S530, determining whether to issue an early warning to the vehicle based on the motion trajectory of the object's bounding box according to determining that the bounding box of the object is not located at the boundary of the corresponding image, may include: determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the position information of the bounding box of the object; determining whether the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds the preset trajectory motion threshold according to determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold; determining whether the motion trajectory of the object is away from the vehicle or close to the vehicle according to determining that the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds the preset trajectory motion threshold; not issuing an early warning to the vehicle according to determining that the motion trajectory of the object is away from the vehicle; and issuing an early warning to the vehicle according to determining that the motion trajectory of the object is close to the vehicle; not issuing an early warning to the vehicle according to determining that the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image does not exceed the preset trajectory motion threshold.

[0086] Similar to the above-mentioned embodiment in which the bounding box of the object is located at the boundary of the corresponding image, when the bounding box of the object is not located at the boundary of the corresponding image, if the distance between the object and the vehicle is less than the preset trajectory motion threshold, it can be determined whether to issue an early warning to the vehicle in combination with the object's motion speed. The object's motion speed can be determined by calculating the displacement change of the object's bounding box in the corresponding image and the previous adjacent image of the corresponding image. In the case where the object's motion speed is relatively large, it can be further determined whether the object's motion trajectory is away from the vehicle or close to the vehicle. If it is away from the vehicle, no early warning may be issued to the vehicle, and if it is close to the vehicle, an early warning may be issued to the vehicle. In the case where the object's motion speed is relatively small, it can be considered that the motion speed is caused by vehicle body shaking or bumping, etc., and will not cause a large collision risk to the vehicle, so no early warning may be issued to the vehicle.

[0087] Similarly, the preset trajectory motion change threshold value can also be any suitable pixel value set according to the requirements. The smaller the preset trajectory motion change threshold value, the earlier the vehicle warning is, thereby reducing the occurrence of vehicle collision accidents. The larger the preset trajectory motion change threshold value, the later the vehicle warning is, thereby avoiding frequent warnings to the vehicle driver.

[0088] When the vehicle body shakes or bumps, it usually causes the bounding box of the object in the corresponding image to change (for example, the change in position information may cause the determined distance between the object and the vehicle to change). However, this change usually does not affect the collision risk factor. By setting a preset trajectory motion change threshold, early and frequent warnings to the vehicle driver in this case can be avoided.

[0089] It will be appreciated that the determination of whether to issue an early warning to the vehicle may also be made in conjunction with the object's motion acceleration. For example, the object's motion acceleration may be further determined, and the vehicle may be issued an early warning when the object's motion acceleration is large, while the vehicle may not be issued an early warning when the object's deceleration is large (i.e., the object is decelerating).

[0090] It will also be understood that the preset width change threshold and / or the preset trajectory motion change threshold can be varied based on the distance between the object and the vehicle. In some examples, the distance between the object and the vehicle is far, and the possibility of a collision is relatively small. In other examples, the distance between the object and the vehicle is close, and the possibility of a collision is relatively increased. By adaptively adjusting the preset width change threshold and / or the preset trajectory motion change threshold according to the distance between the object and the vehicle, the vehicle driver can be warned more flexibly and accurately.

[0091] Figure 6 FIG. 1 is a flow chart showing a method for determining whether to issue a warning to a vehicle based on the motion trajectory of one or more objects according to another exemplary embodiment. Figure 6 As shown, step S140, determining whether to warn the vehicle based on the motion trajectory of one or more objects may include: step S610, determining the position relationship between one or more objects and the vehicle; and step S620, warning the vehicle based on the position relationship.

[0092] By determining the positional relationship between one or more objects and the vehicle, different warning algorithms can be set for the area where the objects are located, thereby providing more accurate warning information, so that the vehicle driver can more clearly understand the cause of the warning and respond to the warning information in a timely manner.

[0093] According to some embodiments of the present disclosure, the positional relationship may include one or more of the following: one or more objects are located in a blind spot of the vehicle; one or more objects are located in an adjacent lane of the vehicle; or one or more objects are located behind the vehicle.

[0094] When one or more objects are located in the blind spot of the vehicle, Blind Spot Detection (BSD) can be enabled to monitor the driver's blind spot in real time and warn the driver when other road users appear in their blind spot.

[0095] When one or more objects are located in the adjacent lane of the vehicle, Lane Changing Warning (LCW) can be activated to monitor the adjacent lane when the vehicle changes lanes and warn the driver if other road users who may collide with the vehicle appear on the side / rear of the vehicle.

[0096] When one or more objects are located behind the vehicle, the Rear Collision Warning (RCW) can be activated to monitor the area behind the vehicle in real time and warn the driver when other road users who may collide with it from behind appear.

[0097] Figure 7 FIG. 7 shows a structural block diagram of a vehicle warning device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the vehicle warning device 700 may include: a capture unit 710, configured to capture multiple images during vehicle driving via a sensor, wherein the sensor is located at the rear of the vehicle; a target detection unit 720, configured to perform target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of one or more bounding boxes; a motion trajectory determination unit 730, configured to determine the motion trajectory of one or more objects based on the one or more bounding boxes and their position information; and a warning unit 740, configured to determine whether to warn the vehicle based on the motion trajectory of the one or more objects.

[0098] Compared with traditional distance measuring sensors, the vehicle warning device according to the embodiment of the present disclosure can capture multiple images during vehicle driving based on visual sensors, and based on this, locate and identify objects behind the vehicle and judge the movement trend. Since target detection and positioning based on image-based visual information and judgment of movement trends are more efficient, and only one image sensor is used to achieve the above operations, it can reduce costs, simplify operations and reduce failure rates.

[0099] According to some embodiments of the present disclosure, the vehicle warning device 700 may further include: a unit configured to perform target detection on each of a plurality of images to obtain categories of one or more objects; and wherein determining the motion trajectory of one or more objects is further based on the categories of one or more objects.

[0100] According to some embodiments of the present disclosure, the target detection unit 720 may include: a target detection subunit, configured to perform target detection on each of a plurality of images based on a neural network model to obtain one or more bounding boxes of one or more objects and location information of the one or more bounding boxes.

[0101] According to some embodiments of the present disclosure, a target detection subunit may include: a unit configured to perform target detection on multiple images based on a student model network, wherein the student model network is obtained by the following operations: a unit training a teacher model network, the teacher model network having a first parameter amount; and obtaining a student model network based on the teacher model network, the student model network having a second parameter amount less than the first parameter amount.

[0102] According to some embodiments of the present disclosure, the target detection subunit may include: a unit configured to perform quantization processing on the neural network model to convert an input data type of the neural network model into a predefined data type.

[0103] According to some embodiments of the present disclosure, the motion trajectory determination unit 730 may include a unit that performs the following operations for each object of one or more objects in each image: determining a confidence level between the object and each object in an image captured before the image; determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence level is greater than a first confidence threshold; determining whether the confidence level is greater than a second confidence threshold based on determining that the confidence level is not greater than the first confidence threshold, wherein the second confidence threshold is less than the first confidence threshold; determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence level is greater than the second confidence threshold; and determining the motion trajectory of the object based on the bounding box of the matched object in multiple images and its position information.

[0104] According to some embodiments of the present disclosure, the vehicle warning device 700 further includes a unit configured to obtain motion information of a sensor, and wherein the motion trajectory determination unit 730 may include: a unit configured to determine, for each of one or more objects in each image, whether the object matches a corresponding object in an image captured before the image based on both confidence and motion information.

[0105] According to some embodiments of the present disclosure, the vehicle warning device 700 may further include: a motion trajectory correction unit configured to correct the motion trajectory of one or more objects.

[0106] According to some embodiments of the present disclosure, the motion trajectory correction unit may include: a unit configured to capture the posture information of the vehicle during driving; and a unit configured to correct the motion trajectory of one or more objects based on the posture information.

[0107] According to some embodiments of the present disclosure, a motion trajectory correction unit may include a unit configured to perform the following operations: for each object of one or more objects, based on determining that a bounding box of the object is not detected in at least one image of a plurality of images, compensate for the position information of the bounding box of the object in at least one image based on the position information of the bounding box of the object detected in a first image and the position information of the bounding box of the object detected in a second image to obtain a corrected motion trajectory of the object, wherein the first image is an image captured before at least one image of the plurality of images, and the second image is an image captured after at least one image of the plurality of images.

[0108] According to some embodiments of the present disclosure, the early warning unit 740 may include a unit configured to perform the following operations: determining, for each of one or more objects, whether a bounding box of the object is located at a boundary of a corresponding image; determining, based on determining that the bounding box of the object is located at the boundary of the corresponding image, whether to warn a vehicle based on a change in the width of the bounding box of the object in the corresponding image; and determining, based on determining that the bounding box of the object is not located at the boundary of the corresponding image, whether to warn the vehicle based on a movement trajectory of the bounding box of the object.

[0109] According to some embodiments of the present disclosure, the position information of one or more bounding boxes may be the midpoint of the lower boundary of the corresponding bounding box, and wherein, for each of the one or more objects, determining whether the bounding box of the object is located at the boundary of the corresponding image may include: determining that the bounding box of the object is located at the boundary of the corresponding image based on determining that at least a portion of the bounding box of the object is located outside the left boundary or the right boundary of the corresponding image; and determining that the bounding box of the object is not located at the boundary of the corresponding image based on determining that the entire bounding box of the object is located in the corresponding image.

[0110] According to some embodiments of the present disclosure, determining whether to issue an early warning to a vehicle based on a change in the width of the object's bounding box in the corresponding image, based on determining that the object's bounding box is located at the boundary of the corresponding image, may include: determining whether the width of the object's bounding box in the corresponding image is less than a preset width threshold; issuing an early warning to the vehicle based on determining that the width of the object's bounding box in the corresponding image is less than the preset width threshold; and determining whether the object's motion trajectory is away from the vehicle or close to the vehicle based on determining that the object's motion trajectory is away from the vehicle, not issuing an early warning to the vehicle; and issuing an early warning to the vehicle when the width of the object's bounding box in the corresponding image is less than the preset width threshold based on determining that the object's motion trajectory is close to the vehicle.

[0111] According to some embodiments of the present disclosure, according to determining that the bounding box of the object is located at the boundary of the corresponding image, determining whether to warn the vehicle based on the width change of the bounding box of the object in the corresponding image may include: determining whether the width of the bounding box of the object in the corresponding image is less than a preset width threshold; according to determining that the width of the bounding box of the object in the corresponding image is less than the preset width threshold: determining whether the width change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds a preset width change threshold; according to determining that the width change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds a preset width change threshold: determining whether the motion trajectory of the object is away from the vehicle or close to the vehicle; according to determining that the motion trajectory of the object is away from the vehicle, not warning the vehicle; and according to determining that the motion trajectory of the object is close to the vehicle, warning the vehicle; according to determining that the width change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image does not exceed the preset width change threshold, not warning the vehicle.

[0112] According to some embodiments of the present disclosure, the motion trajectory of the object is determined to be away from the vehicle based on the width of the bounding box of the object in the corresponding image being increased, and the motion trajectory of the object is determined to be close to the vehicle based on the width of the bounding box of the object in the corresponding image being decreased.

[0113] According to some embodiments of the present disclosure, determining whether to issue an early warning to the vehicle based on the motion trajectory of the object's bounding box, based on determining that the bounding box of the object is not located at the boundary of the corresponding image, may include: determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the so-called position information of the bounding box of the object; issuing an early warning to the vehicle based on determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold; and determining whether the motion trajectory of the object is away from the vehicle or close to the vehicle based on determining that the distance between the object and the vehicle is not less than the preset trajectory motion threshold; not issuing an early warning to the vehicle based on determining that the motion trajectory of the object is away from the vehicle; and issuing an early warning to the vehicle based on determining that the motion trajectory of the object is close to the vehicle when the distance exceeds the preset trajectory motion threshold.

[0114] According to some embodiments of the present disclosure, determining whether to issue an early warning to the vehicle based on the motion trajectory of the object's bounding box according to determining that the bounding box of the object is not located at the boundary of the corresponding image may include: determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the position information of the bounding box of the object; determining whether the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds a preset trajectory motion change threshold according to determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold; determining whether the motion trajectory of the object is away from the vehicle or close to the vehicle according to determining that the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image exceeds the preset trajectory motion change threshold; not issuing an early warning to the vehicle according to determining that the motion trajectory of the object is away from the vehicle; and issuing an early warning to the vehicle according to determining that the motion trajectory of the object is close to the vehicle; not issuing an early warning to the vehicle according to determining that the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image does not exceed the preset trajectory motion change threshold.

[0115] According to some embodiments of the present disclosure, the preset width change threshold and / or the preset trajectory movement change threshold may vary based on the distance between the object and the vehicle.

[0116] According to some embodiments of the present disclosure, the warning unit 740 may include: a unit configured to determine a positional relationship between one or more objects and a vehicle; and a unit configured to warn the vehicle based on the positional relationship.

[0117] According to some embodiments of the present disclosure, the positional relationship includes one or more of the following: one or more objects are located in a blind spot of the vehicle; one or more objects are located in an adjacent lane of the vehicle; or one or more objects are located behind the vehicle.

[0118] According to some embodiments of the present disclosure, the vehicle may be a two-wheeled vehicle or a unicycle.

[0119] It should be understood that Figure 7 The various units of the vehicle warning device 700 shown in FIG. Figure 1 and Figure 4 The various steps in the described vehicle warning methods 100 and 400 correspond to each other. Therefore, the operations, features and advantages described above for the methods 100 and 400 are also applicable to the vehicle warning device 700 and the units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0120] Yet another aspect of the present disclosure may include an electronic device, which may include a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the above-mentioned vehicle warning method.

[0121] Yet another aspect of the present disclosure may include a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned vehicle warning method when executed by a processor.

[0122] Yet another aspect of the present disclosure may include a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned vehicle warning method are implemented.

[0123] Reference Figure 8 , a computing device 800 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The computing device 800 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, an in-vehicle computer, an access control system, an attendance device, or any combination thereof. The above-mentioned vehicle warning device can be implemented in whole or in part by the computing device 800 or a similar device or system. Although the computing device 800 represents an example of several types of computing platforms, the computing device 800 can include more than Figure 8 Greater or fewer elements and / or different arrangements of elements may be shown, and the scope of the claimed subject matter is not limited in these respects.

[0124] In some embodiments, the computing device 800 may include an element connected to or in communication with the bus 802 (possibly via one or more interfaces). For example, the computing device 800 may include a bus 802, one or more processors 804, one or more input devices 806, and one or more output devices 808. The one or more processors 804 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 806 may be any type of device that can input information to the computing device 800, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote control. The output device 808 may be any type of device that can present information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 800 may also include or be connected to a non-transitory storage device 810. The non-transitory storage device may be any storage device that is non-transitory and can implement data storage, and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape, or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory, and / or any other memory chip or box, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 810 may be detachable from the interface. The non-transitory storage device 810 embodies one or more non-transitory computer-readable media on which a program including instructions is stored, which, when executed by one or more processors of the computing device 800, causes the computing device 800 to perform the above-mentioned vehicle warning method 100, 400 and its variants. The computing device 800 may also include a communication device 812. The communication device 812 can be any type of device or system that enables communication with an external device and / or with a network, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0125] In some embodiments, computing device 800 may also include working memory 814, which may be any type of memory that can store programs (including instructions) and / or data useful for operation of processor 804, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0126] Software elements (programs) may be located in the working memory 814, including but not limited to an operating system 816, one or more applications 818, drivers and / or other data and codes. Instructions for executing the above methods and steps may be included in one or more applications 818, and the electronic circuit of the above vehicle warning device may be implemented by reading and executing the instructions of one or more applications 818 by the processor 804. The executable code or source code of the instructions of the software element (program) may be stored in a non-transitory computer-readable storage medium (such as the above-mentioned storage device 810), and may be stored in the working memory 814 (possibly compiled and / or installed) when executed. The executable code or source code of the instructions of the software element (program) may also be downloaded from a remote location.

[0127] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0128] It should also be understood that the aforementioned method can be implemented in a server-client mode. For example, the client can use a camera to collect image data and send the image data to the server for subsequent processing. The client can also perform a portion of the processing in the aforementioned method and send the processed data to the server. The server can receive data from the client and execute the aforementioned method or another portion of the aforementioned method, and return the execution result to the client. The client can receive the execution result of the method from the server and present it to the user, for example, through an output device.

[0129] It should also be understood that the components of computing device 800 can be distributed over a network. For example, some processing can be performed using one processor, while other processing can be performed by another processor remote from the one processor. Other components of computing device 800 can also be similarly distributed. In this way, computing device 800 can be understood as a distributed computing system that performs processing on multiple processors at multiple locations.

[0130] Some exemplary aspects of the disclosure are described below.

[0131] Aspect 1. A vehicle early warning method, comprising:

[0132] capturing a plurality of images during travel of the vehicle via a sensor, the sensor being located at a rear portion of the vehicle;

[0133] Performing object detection on each of the plurality of images to obtain one or more bounding boxes of one or more objects and location information of the one or more bounding boxes;

[0134] Determining a motion trajectory of the one or more objects based on the one or more bounding boxes and their position information; and

[0135] Determine whether to issue an early warning to the vehicle based on the movement trajectory of the one or more objects.

[0136] Aspect 2. The method according to aspect 1 further comprises:

[0137] Performing target detection on each of the plurality of images to obtain categories of the one or more objects; and

[0138] Wherein, determining the motion trajectory of the one or more objects is further based on the category of the one or more objects.

[0139] Aspect 3. The method according to aspect 1, wherein performing target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes comprises:

[0140] Perform object detection on each of the multiple images based on the neural network model to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes.

[0141] Aspect 4. The method according to aspect 3, wherein performing target detection on each of the multiple images based on the neural network model comprises:

[0142] Performing object detection on the multiple images based on the student model network, wherein the student model network is obtained by the following operations:

[0143] training a teacher model network, the teacher model network having a first number of parameters; and

[0144] A student model network is obtained based on the teacher model network, wherein the student model network has a second parameter amount that is smaller than the first parameter amount.

[0145] Aspect 5. The method according to aspect 3, wherein performing target detection on the multiple images based on a neural network model comprises:

[0146] The neural network model is quantized to convert an input data type of the neural network model into a predefined data type.

[0147] Aspect 6. The method according to aspect 2, wherein determining the motion trajectory of the one or more objects based on the one or more bounding boxes and their position information comprises:

[0148] For each of the one or more objects in each of the images:

[0149] determining a confidence level between the object and each object in an image captured before the image;

[0150] Determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence is greater than a first confidence threshold;

[0151] Determining whether the confidence level is greater than a second confidence level threshold based on determining that the confidence level is not greater than a first confidence level threshold, wherein the second confidence level threshold is less than the first confidence level threshold;

[0152] Determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence level is greater than the second confidence threshold; and

[0153] Based on the bounding boxes of the object matched in the multiple images and the position information thereof, a motion trajectory of the object is determined.

[0154] Aspect 7. The method according to aspect 6 further comprises acquiring motion information of the sensor; and

[0155] Wherein, based on the one or more bounding boxes and their position information, determining the motion trajectory of the one or more objects further includes:

[0156] For each of the one or more objects in each of the images, it is determined based on both the confidence level and the motion information that the object matches a corresponding object in an image captured before the image.

[0157] Aspect 8. The method according to any one of aspects 1 to 7, further comprising:

[0158] The motion trajectory of the one or more objects is corrected.

[0159] Aspect 9. The method according to aspect 8, wherein correcting the motion trajectory of the one or more objects comprises:

[0160] Capturing the position information of the vehicle during driving; and

[0161] The motion trajectory of the one or more objects is corrected based on the posture information.

[0162] Aspect 10. The method according to aspect 8, wherein correcting the motion trajectory of the one or more objects comprises:

[0163] For each of the one or more objects, according to determining that a bounding box of the object is not detected in at least one image of the multiple images, based on the position information of the bounding box of the object detected in the first image and the position information of the bounding box of the object detected in the second image, compensating for the position information of the bounding box of the object in the at least one image to obtain a corrected motion trajectory of the object,

[0164] The first image is an image captured before the at least one image among the multiple images, and the second image is an image captured after the at least one image among the multiple images.

[0165] Aspect 11. The method according to any one of aspects 1 to 7, wherein determining whether to issue a warning to the vehicle based on the movement trajectory of the one or more objects comprises:

[0166] For each of the one or more objects, determining whether a bounding box of the object is located at a boundary of the corresponding image;

[0167] According to determining that the bounding box of the object is located at the boundary of the corresponding image, determining whether to issue a warning to the vehicle based on a width change of the bounding box of the object in the corresponding image; and

[0168] According to determining that the bounding box of the object is not located at the boundary of the corresponding image, it is determined whether to issue a warning to the vehicle based on the motion trajectory of the bounding box of the object.

[0169] Clause 12. The method according to clause 11, wherein the position information of the one or more bounding boxes is the midpoint of the lower border of the corresponding bounding box, and

[0170] Wherein, for each of the one or more objects, determining whether a bounding box of the object is located at a boundary of the corresponding image comprises:

[0171] Determining that the bounding box of the object is located at a boundary of the corresponding image based on determining that at least a portion of the bounding box of the object is located outside a left boundary or a right boundary of the corresponding image; and

[0172] According to determining that the entirety of the bounding box of the object is located in the corresponding image, it is determined that the bounding box of the object is not located at a boundary of the corresponding image.

[0173] Aspect 13. The method according to aspect 11, wherein, according to determining that the bounding box of the object is located at the boundary of the corresponding image, determining whether to issue a warning to the vehicle based on a change in the width of the bounding box of the object in the corresponding image comprises:

[0174] Determining whether a width of a bounding box of the object in the corresponding image is less than a preset width threshold;

[0175] According to determining that the width of the bounding box of the object in the corresponding image is less than the preset width threshold:

[0176] issuing an early warning to the vehicle; and

[0177] According to determining that the width of the bounding box of the object in the corresponding image is not less than the preset width threshold:

[0178] Determining whether the object's motion trajectory is moving away from the vehicle or approaching the vehicle;

[0179] Based on determining that the movement trajectory of the object is away from the vehicle, no warning is issued to the vehicle; and based on determining that the movement trajectory of the object is close to the vehicle, a warning is issued to the vehicle when the width of the bounding box of the object in the corresponding image is less than the preset width threshold.

[0180] Aspect 14. The method according to aspect 11, wherein, according to determining that the bounding box of the object is located at the boundary of the corresponding image, determining whether to issue a warning to the vehicle based on a width change of the bounding box of the object in the corresponding image comprises:

[0181] Determining whether a width of a bounding box of the object in the corresponding image is less than a preset width threshold;

[0182] According to determining that the width of the bounding box of the object in the corresponding image is less than the preset width threshold:

[0183] Determine whether a width change of the bounding box of the object in the corresponding image and a previous adjacent image of the corresponding image exceeds a preset width change threshold;

[0184] According to determining that the bounding box of the object is located between the corresponding image and the previous image of the corresponding image

[0185] The width variation in the adjacent image exceeds the preset width variation threshold:

[0186] Determining whether the object's motion trajectory is moving away from the vehicle or approaching the vehicle;

[0187] According to determining that the movement trajectory of the object is far away from the vehicle, not issuing a warning to the vehicle; and

[0188] According to determining that the movement trajectory of the object is close to the vehicle, issuing an early warning to the vehicle;

[0189] According to determining that the width change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image does not exceed the preset width change threshold, no warning is issued to the vehicle.

[0190] Aspect 15. A method according to Aspect 13 or 14, wherein the movement trajectory of the object is determined to be away from the vehicle based on the width of the bounding box of the object in the corresponding image being determined to be larger, and wherein the movement trajectory of the object is determined to be close to the vehicle based on the width of the bounding box of the object in the corresponding image being determined to be smaller.

[0191] Aspect 16. The method according to aspect 11, wherein, according to determining that the bounding box of the object is not located at the boundary of the corresponding image, determining whether to issue a warning to the vehicle based on the motion trajectory of the bounding box of the object comprises:

[0192] Determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the position information of the bounding box of the object;

[0193] According to determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold:

[0194] issuing an early warning to the vehicle; and

[0195] According to determining that the distance between the object and the vehicle is not less than the preset trajectory motion threshold:

[0196] Determining whether the object's motion trajectory is moving away from the vehicle or approaching the vehicle;

[0197] Based on determining that the movement trajectory of the object is far away from the vehicle, no warning is issued to the vehicle; and based on determining that the movement trajectory of the object is close to the vehicle, a warning is issued to the vehicle when the distance exceeds the preset trajectory movement threshold.

[0198] Aspect 17. The method according to aspect 11, wherein, according to determining that the bounding box of the object is not located at the boundary of the corresponding image, determining whether to issue a warning to the vehicle based on the motion trajectory of the bounding box of the object comprises:

[0199] Determining whether the distance between the object and the vehicle is less than a preset trajectory motion threshold based on the position information of the bounding box of the object;

[0200] According to determining that the distance between the object and the vehicle is less than the preset trajectory motion threshold:

[0201] Determine whether a displacement change of the bounding box of the object in the corresponding image and a previous adjacent image of the corresponding image exceeds a preset trajectory motion change threshold;

[0202] According to determining that the bounding box of the object is located between the corresponding image and the previous image of the corresponding image

[0203] The displacement change in the adjacent image exceeds the preset trajectory motion change threshold:

[0204] Determining whether the object's motion trajectory is moving away from the vehicle or approaching the vehicle;

[0205] According to determining that the movement trajectory of the object is far away from the vehicle, not issuing a warning to the vehicle; and

[0206] According to determining that the movement trajectory of the object is close to the vehicle, issuing an early warning to the vehicle;

[0207] According to determining that the displacement change of the bounding box of the object in the corresponding image and the previous adjacent image of the corresponding image does not exceed the preset trajectory motion change threshold, no warning is issued for the vehicle.

[0208] Aspect 18. The method according to Aspect 14 or 17, wherein the preset width change threshold and / or the preset trajectory movement change threshold varies based on the distance between the object and the vehicle.

[0209] Aspect 19. The method according to any one of aspects 1 to 7, wherein determining whether to issue a warning to the vehicle based on the movement trajectory of the one or more objects comprises:

[0210] determining a positional relationship of the one or more objects to the vehicle; and

[0211] The vehicle is warned based on the positional relationship.

[0212] Aspect 20. The method according to aspect 19, wherein the position relationship comprises one or more of the following:

[0213] the one or more objects being located in a blind spot of the vehicle;

[0214] The one or more objects are located in an adjacent lane of the vehicle; or

[0215] The one or more objects are located behind the vehicle.

[0216] Aspect 21. The method according to any one of aspects 1 to 7, wherein the vehicle is a two-wheeled vehicle or a unicycle.

[0217] Aspect 22. A vehicle warning device, comprising:

[0218] a capturing unit configured to capture a plurality of images during the driving of the vehicle via a sensor, wherein the sensor is located at a rear portion of the vehicle;

[0219] an object detection unit configured to perform object detection on each of the plurality of images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes;

[0220] a motion trajectory determining unit configured to determine the motion trajectory of the one or more objects based on the one or more bounding boxes and their position information; and

[0221] The warning unit is configured to determine whether to issue a warning to the vehicle based on the movement trajectory of the one or more objects.

[0222] Aspect 23. An electronic device comprising:

[0223] a memory, a processor and a computer program stored on the memory,

[0224] The processor is configured to execute the computer program to implement the steps of the method according to any one of aspects 1 to 21.

[0225] Aspect 24. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 21 are implemented.

[0226] Aspect 25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of aspects 1 to 21 are implemented.

[0227] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A vehicle early warning method, comprising: capturing a plurality of images during travel of the vehicle via a sensor, the sensor being located at a rear portion of the vehicle; Performing object detection on each of the plurality of images to obtain one or more bounding boxes of one or more objects and location information of the one or more bounding boxes; Determining a motion trajectory of the one or more objects based on the one or more bounding boxes and their position information; as well as Determine whether to issue an early warning to the vehicle based on the movement trajectory of the one or more objects.

2. The method according to claim 1, further comprising: Performing object detection on each of the multiple images to obtain categories of the one or more objects; and Wherein, determining the motion trajectory of the one or more objects is further based on the category of the one or more objects.

3. The method according to claim 1, wherein: Performing target detection on each of the multiple images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes includes: Perform object detection on each of the multiple images based on the neural network model to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes.

4. The method according to claim 3, wherein: Performing target detection on each of the multiple images based on the neural network model includes: Performing object detection on the multiple images based on the student model network, wherein the student model network is obtained by the following operations: training a teacher model network, the teacher model network having a first number of parameters; and A student model network is obtained based on the teacher model network, wherein the student model network has a second parameter amount that is smaller than the first parameter amount.

5. The method according to claim 3, wherein: Performing target detection on the multiple images based on the neural network model includes: The neural network model is quantized to convert an input data type of the neural network model into a predefined data type.

6. The method according to claim 2, wherein: Determining the motion trajectory of the one or more objects based on the one or more bounding boxes and their position information includes: For each of the one or more objects in each of the images: determining a confidence level between the object and each object in an image captured before the image; Determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence is greater than a first confidence threshold; Determining whether the confidence level is greater than a second confidence level threshold based on determining that the confidence level is not greater than a first confidence level threshold, wherein the second confidence level threshold is less than the first confidence level threshold; Determining that the object matches a corresponding object in an image captured before the image based on determining that the confidence level is greater than the second confidence threshold; and Based on the bounding boxes of the object matched in the multiple images and the position information thereof, a motion trajectory of the object is determined.

7. A vehicle warning device, comprising: a capturing unit configured to capture a plurality of images during the driving of the vehicle via a sensor, wherein the sensor is located at a rear portion of the vehicle; an object detection unit configured to perform object detection on each of the plurality of images to obtain one or more bounding boxes of one or more objects and position information of the one or more bounding boxes; a motion trajectory determining unit, configured to determine the motion trajectory of the one or more objects based on the one or more bounding boxes and position information thereof; as well as The warning unit is configured to determine whether to issue a warning to the vehicle based on the movement trajectory of the one or more objects.

8. An electronic device comprising: a memory, a processor and a computer program stored on the memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images