Automatic detection of lost cargo / generic objects in video sequence of road

Abstract

A control unit for automatic detection of lost cargo / generic objects in a video sequence of a road in a vehicle. The control unit includes a camera mounted outside the vehicle, for capturing a real-time video sequence of the road, and an intelligent model constructed and trained through a main dataset of objects having a plurality of frame / data points of the road and a marker in each frame / data point. The control unit fine-tunes the intelligent model by training with the synthetic data set. A control unit generates candidate patches / tags from each of the frames / data points of the synthetic data set and the master data set from the semantic segmentation stage. The control unit receives a plurality of textual cues related to tags of the main data set and the composite data set. The control unit identifies at least one lost cargo / generic object in the real-time video sequence of the road by using the multi-modal base model.

Description

Technical Field

[0001] This invention relates to the automatic detection of missing goods / general objects in video sequences of roads. The automatic detection is performed by a control unit. Background Technology

[0002] While detecting known classes of objects such as vehicles and pedestrians in road images is a well-studied problem in the literature, it should be noted that the problem of generic (lost cargo) objects with unknown classes has not been fully explored. Although often small, lost cargo objects still pose a danger to autonomous driving applications because they are unsafe to drive over.

[0003] U.S. Patent 10,127,464 discloses a system that can be installed in a vehicle to provide object detection near the vehicle. The system includes a camera operatively attached to a processor. The camera is externally mounted at the rear of the vehicle. The camera's field of view is substantially in the forward direction along the side of the vehicle. Multiple image frames are captured from the camera. The yaw of the vehicle can be input, or the yaw can be calculated from the image frames. A corresponding portion of the image frames is selected in response to the yaw of the vehicle. The image frames are processed to detect objects in the selected portion of the image frames. Attached Figure Description

[0004] Figure 1 An automatic detection control unit for lost cargo / general objects in a video sequence of a road is shown according to an embodiment of the present invention; and

[0005] Figure 2 A method for automatically detecting missing goods / general objects in a video sequence of a road, according to the present invention, is illustrated. Detailed Implementation

[0006] Figure 1A control unit for automatic detection of missing cargo / general objects in a video sequence of a road in a vehicle, according to an embodiment of the present invention, is illustrated. The control unit 12 includes a camera 21 mounted externally to a vehicle 10 for capturing real-time video sequences of the road, and an intelligent model 18 constructed and trained from a main dataset 14 having multiple frames / data points of the road and labeled objects in each frame / data point. The control unit 12 fine-tunes the intelligent model 18 by training with a synthetic dataset 16. The control unit 12 generates candidate patches / labels from each of the frames / data points from the synthetic dataset 16 and the main dataset 14 from the semantic segmentation stage. The control unit 12 receives multiple text cues 24 associated with the labels in the main dataset 14 and the synthetic dataset 16. The control unit 12 identifies the presence of at least one missing cargo / general object 11 in the real-time video sequence of the road by using a multimodal base model, which is a Visual Language Foundation Model (VLFM) 17, and by using the received multiple text cues 24.

[0007] The construction of the framework, including the control unit and the intelligent model, is explained in further detail. The control unit 12 includes an intelligent model 18 and a multimodal base model 22. The intelligent model 18 is constructed and trained using a main dataset 14 comprising multiple data points / frames. In this paper, a data point / frame refers to multiple frames of different segments of a road. Each data point / frame undergoes a semantic segmentation process to identify objects 11 in each frame / data point. The identified objects in each frame / data point are referred to as labels. Therefore, the trained intelligent model 18 with the main labeled dataset 14 is fine-tuned by providing additional data in the form of a synthetic dataset 16. The synthetic data 16 in this paper refers to the additional frames along with the labels identified in each frame / data point, which are new and not present in the list of labels in the main dataset 14. The synthetic data 16 includes different types of generic / missing cargo objects 11 (synthetically pasted onto a new background) and their corresponding labels.

[0008] For example, if the intelligent model 18 is pre-trained using a main training dataset 14 containing thousands of pre-trained images with multiple labels (identified objects), then a synthetic dataset 16 containing approximately thousands of labeled frames / data points is used to fine-tune the intelligent model 18, and these together undergo a semantic segmentation process in the segmentation module 20 for fine-tuning the intelligent model 18. The intelligent model 18 further receives frames / data points in real-time from a camera 21 mounted on the vehicle 10. Let approximately 50 frames be received from the camera 21, and then sent to the segmentation module 20 for inference results from the semantic segmentation model. In another embodiment of the invention, the real-time input from the camera 21 can be a single frame / data point undergoing a semantic segmentation process in the segmentation module 20.

[0009] Control unit 12 generates candidate patches / tags as output from the received data points / frames. The candidate patches / tags are provided to multimodal base model 22 for further filtering. Multimodal base model 22 performs image retrieval using text cues 24 through cue engineering techniques. Control unit 12 structures the input text cues 24 using cue engineering techniques to accurately interpret at least one missing cargo / general object 11. Control unit 12 identifies at least one missing cargo / general object 11 in the real-time video sequence of the road based on quantitative parameters such as accuracy parameters and recall parameters. However, it should be noted that quantitative parameters can be any other parameters, but are not limited to those described above.

[0010] Control unit 12 calculates an accuracy parameter that indicates the correct number of identified lost cargo / general object 11, and a recall parameter that indicates the accuracy of the multimodal base model 22 in recalling classes from frames / data points. Candidate patches from smart model 18, along with text prompts 24, are provided to multimodal base model 22 for real-time identification of at least one lost cargo / general object 11 in frames of a video sequence received from camera 21. Control unit 12 generates an alarm 25 in vehicle 10 when at least one lost cargo / general object 11 is detected in that particular frame / data point and the location of the lost cargo / general object 11 on the road.

[0011] Figure 2 A method for automatically detecting lost goods / general objects in a video sequence of a road according to the present invention is illustrated. The method uses a control unit 12 to detect lost goods / general objects 11. The control unit 12 includes a camera 21 mounted externally to a vehicle 10 for capturing real-time video sequences of the road, and an intelligent model 18 constructed and trained from a master dataset 14 having multiple frames / data points of the road and labeled objects in each frame / data point. In step S1, the intelligent model 18 is fine-tuned by training with a synthetic dataset 16. In step S2, candidate patches / labels are generated from each of the data points / frames from the synthetic dataset 16 and the master dataset 14 from the semantic segmentation stage. In step S3, multiple text cues 24 associated with the labels in the master dataset 14 and the synthetic dataset 16 are received. In step S4, at least one lost goods / general object 11 in the real-time video sequence of the road is identified by using a multimodal base model 22 and the multiple text cues 24.

[0012] This method is explained in detail. Lost cargo / general objects 11, though often small, still pose a danger to autonomous driving applications because it is unsafe for them to pass by. These objects 11 include, typically, stationary fallen bicycles, pylons, dead animals, stones or rocks, trash cans, boxes, etc. Detecting such lost cargo / general objects on the road, such as unclaimed and potentially suspicious packages, injured people or animals, is a necessary application in autonomous driving systems. A detection method is needed to identify lost cargo / general objects 11 that are not restricted by class type. Note that most computer vision object detection methods are trained on a wide variety of objects and networks, specifically on driving datasets (detection or segmentation), and can only correctly identify known objects, but cannot easily detect or segment unknown objects without explicit training for this task.

[0013] This invention provides a solution to the problem. Because the amount of real-world lost cargo / general object data is extremely small, this invention provides an efficient framework to utilize synthetic data, VLFM, and cue engineering to detect hazardous objects in a computationally efficient manner. Conventional framework only uses image, video, or stereo data and derives distinct hand-crafted features for each type of object. No prior art currently uses a Visual Language Foundation Model (VLFM) for robust lost cargo / general object detection methods. This invention solves this problem and describes a complete end-to-end framework for solving it.

[0014] This invention comprises a unified hardware and software solution for automatically identifying lost cargo / general objects 11 on a road using a pre-trained multimodal base model 22, which performs filtering on candidate patches based on text prompts 24. The invention employs an artificial intelligence approach that generates several candidate patches / labels from images / frames / data points for potential identification of lost cargo / general objects 11, and then uses a filtering step that uses the base model 22 to obtain only the correct patches (lost cargo patches). This benefits from reduced computational costs because the base model 22 does not need to run on the entire image / frame / data point set.

[0015] A camera 21 is mounted at a specific location on the vehicle 10. An edge computing processor (control unit 12) receives frames from the camera 21 as input. The captured frames are correlated with the road scene captured by the camera 21 from the moving vehicle 10. The captured images are sent to a smart model 18, which has been deployed on edge computing software, for inference.

[0016] Prior to this, a large amount of data had been constructed and trained using multiple frames from different road segments and corresponding labels for each of these frames. Labels refer to different types of objects present in each frame. This data is referred to as the main dataset. Labels of the same type are grouped together into classes, and the dataset includes different classes. For example, if dataset 14 includes 1000 frames / data points and 10 labels for objects / regions, those 10 labels are classified into multiple classes based on class belongingness. This classification process is known in the prior art.

[0017] Furthermore, the intelligent model 18 is fine-tuned using a synthetic dataset 16 that includes frames / data points different from those in the main dataset. This includes a pre-trained model installed on an edge device. In addition, the intelligent model 18 is provided with frames captured in real-time from the camera 21. All data points / frames (received from the main dataset 14, the synthetic dataset 16, and the camera 21) are provided to the intelligent model 18, and semantic segmentation techniques are applied to all of these frames. Therefore, the intelligent model 18 provides candidate patches / labels as output following the semantic segmentation process that occurs in the segmentation module 20 for all of the aforementioned frames. These candidate patches are then provided to the multimodal base model 22.

[0018] The multimodal base model 22 also receives text prompts 24 from the user during the pre-training process. The multimodal base model 22 includes the text prompts 24 and candidate patches from the intelligent model 18. Candidate patches are further filtered by using the multimodal base model 22 as a post-processing step. Note that the control unit 12 does not use the base model 22 across the entire frame, as this is computationally expensive, and only passes candidate patches as input. This invention does not require training the pre-trained base model 22 (which has already been trained on a large number of classes) and can be deployed on an edge computing device to be placed in the vehicle 10. The pre-trained multimodal base model 22 performs filtering on candidate patches based on the text prompts 24. The control unit 12 uses an artificial intelligence method that generates several candidate patches from the frame for potential identification of lost cargo, and then uses the filtering step, which uses the base model, to obtain only the correct patches.

[0019] The accuracy of missing goods / general objects detection using a multimodal base model 22 that employs text cues 24 for image retrieval largely depends on cue engineering techniques. Cue engineering techniques involve appropriately structuring the input text cues 24 so that they can be accurately interpreted by a generative artificial intelligence model 18. The detection of missing goods / general objects 11 depends on many factors, such as lighting, camera type, distance from the camera, etc. It should be noted that false positives (FP) or false negatives (FN) should be avoided. At least one missing goods / general object 11 in a real-time video sequence of a road is identified based on quantitative parameters such as accuracy parameters and call parameters.

[0020] Accuracy and recall parameters are two important quantitative measures used for image classification or missing goods / general objects detection. Control unit 12 determines the accuracy (P) parameter based on the correct number of missing goods / general objects 11 detected. The accuracy parameter intuitively provides information about how accurate the base model 22 is, i.e., how many of the total detected missing goods / general objects were actually correctly identified. Therefore, it is the ratio between true certainty (TP) and the total number predicted. Control unit 12 determines the accuracy parameter using the following formula.

[0021] P = TP / (TP + FP)

[0022] The invocation parameter is a measure of how well the base model 22 invokes classes from images; that is, the number of missing cargo / general objects 11 that the base model 22 can accurately detect out of the total number of missing cargo objects 11 in the input frame. The control unit 12 uses the ratio between the total number of true affirmatives and the total number of ground truth values ​​to determine the invocation parameter.

[0023] R = TP / (TP + FN)

[0024] For most classification or detection problems, there is a trade-off between accuracy and recall. For the preprocessing step, this invention emphasizes that high recall and low accuracy are preferred because the control unit 12 will have a smaller number of missed detections. Since missing cargo / general objects 11 are rare, this invention uses cut-and-paste data augmentation to enhance the samples of missing cargo / general object 11 frames to fine-tune the intelligent model 18, one such intelligent model 18 being the DeepLab semantic segmentation model. The candidate patches generated as output from the intelligent model 18 are further filtered out using a multimodal base model 22 as a post-processing step.

[0025] Note that the present invention does not use the base model 22 for the entire image, as it is computationally expensive, and only candidate patches are passed as input. The output of the base model 22 issues a detection of the missing cargo / general object 11 (with location information from the GPS) if it is encountered. The control unit 12 warns of the presence of the missing cargo / general object 11 by displaying the detection area on the road, enabling the driver to avoid a collision. By using an object bounding box mapped to the real location using GPS, the control unit 12 displays the location of the missing cargo / general object 11 to the user of the vehicle 10.

[0026] Using a prompting process, control unit 12 creates a database of text prompts 24, which selects all possible candidate pieces (high recall, low precision). False positives could be vehicles or road markings (known class) that are considered "normal" and do not warn of lost cargo / general object 11. This invention uses minimal hardware components to identify lost cargo / general object 11. The hardware components involve a set of sensors for measurement, held on vehicle 10, including one or more built-in LED lights and camera sensors 21, an alarm unit 25, and a visual display (not shown) for the identified lost cargo / general object 11. This arrangement provides a low-cost and efficient solution.

[0027] This invention includes an edge computing artificial intelligence method with two phases: a deep learning segmentation model 18 and a multimodal visual language base model 22. Both deployed models have been trained on thousands of dataset images of various objects. Note that the real-world data related to the missing cargo / general object 11 is extremely small, and the aforementioned framework helps to include all possible candidate patches.

[0028] It should be understood that the embodiments described above are merely illustrative and do not limit the scope of the invention. Many such embodiments, as well as other modifications and variations of the embodiments described in the description, are contemplated. The scope of the invention is limited only by the scope of the claims.

Claims

1. A control unit (12) for automatic detection of missing goods / general objects (11) in a video sequence of a road in a vehicle (10), said control unit comprising: - a camera (21) mounted externally on the vehicle (10) for capturing real-time video sequences of a road; - an intelligent model (18) built and trained from a main dataset (14) having a plurality of frames / data points of a road and objects labeled in each frame / data point; characterized in that the control unit (12) is adapted to: - fine-tune the intelligent model (18) by training with a synthetic dataset (16); - generate candidate patches / labels from each of the frames / data points of the main dataset (14) and the synthetic dataset (16) from a semantic segmentation phase; - receive a plurality of textual cues (24) related to the labels of the main dataset (14) and the synthetic dataset (16); - identify at least one missing goods / general object (11) in the real-time video sequences of a road by using a multimodal base model (22) and the received plurality of textual cues (24).

2. The control unit (12) according to claim 1, wherein The synthetic dataset (16) refers to a set of data points / frames having data labeled differently from the set of data points / frames of the main dataset (14).

3. The control unit (12) according to claim 1, wherein The data points / frames of the main dataset (14) and the synthetic dataset (16) undergo a semantic segmentation technique for fine-tuning the intelligent model (18).

4. The control unit (12) according to claim 1, wherein The multimodal base model (22) uses the textual cues (24) for image retrieval by using a cue engineering technique.

5. The control unit (12) according to claim 4, wherein The control unit (12) is adapted to structure the input textual cues (24) by the cue engineering technique to accurately interpret the at least one missing goods / general object (11).

6. The control unit (12) according to claim 1, wherein The identification of the at least one missing goods / general object (11) in the real-time video sequences of a road is based on quantitative parameters such as precision parameters, recall parameters.

7. The control unit (12) according to claim 6, wherein The precision parameters indicate the correct number of missing goods / general objects (11) identified and the recall parameters indicate the accuracy of the multimodal base model (22) to recall classes from frames / data points.

8. The control unit (12) according to claim 1, wherein The candidate patches from the intelligent model (18) are provided to the multimodal base model (22) along with the textual cues (24) for identifying in real-time the at least one missing goods / general object (11) in the frames of the video sequence received from the camera (21).

9. The control unit (12) according to claim 1, wherein When the at least one missing goods / general object (11) is detected in that particular frame / data point and the location of the missing goods / general object (11) on the road, an alert 25 is generated in the vehicle (10) by the control unit (12).

10. A method for automatic detection of missing goods / general objects (11) in a video sequence of a road in a vehicle (10) by a control unit (12), wherein the control unit (12) comprises a camera (21) mounted externally on the vehicle (10) for capturing real-time video sequences of a road and an intelligent model (18) built and trained from a main dataset (14) having a plurality of frames / data points of a road and objects labeled in each frame / data point; characterized in that said method comprising the steps of: - fine-tuning the intelligent model (18) by training with the synthetic dataset (16); - generating a candidate patch / label from each of the data points / frames from the synthetic dataset (16) and the main dataset (14) from the semantic segmentation phase; - receiving a plurality of textual cues (24) related to the labels of the main dataset (14) and the synthetic dataset (16); - identifying at least one missing item / generic object (11) in a real-time video sequence of a road by using a multi-modal base model (22) and by the received plurality of textual cues (24).

Images