Leftover article detection method, model training method, and storage medium

Abstract

This application provides a method for detecting abandoned items, a model training method, and a storage medium. The method includes: acquiring cabin images from multiple cameras within the cabin when a passenger exit signal is detected; inputting the cabin images into a pre-trained item detection model to determine a list of cabin item categories including item location and category information; determining whether the list of cabin item categories includes abandoned items based on pre-set abandoned item judgment rules; and outputting an abandoned item reminder message when the abandoned item is included. This application achieves accurate identification and location of cabin items, preventing items from being left behind and ensuring the safety of user property.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for detecting abandoned items, a model training method, and a storage medium. Background Technology

[0002] With the rise of ride-sharing, car-hailing, driverless taxis, and automated parking, the phenomenon of passengers leaving personal belongings in vehicles is becoming increasingly frequent. For example, it's common for users to leave behind items such as phones, wallets, water bottles, documents, and umbrellas after exiting the vehicle. This not only concerns the safety of users' property but also leads to increased customer service costs, damage to brand image, and even legal liabilities for operators. Therefore, how to promptly detect and alert passengers to leftover items before the vehicle departs has become a critical issue that intelligent cockpit systems urgently need to address. Summary of the Invention

[0003] This application provides a method for detecting abandoned items, a model training method, and a storage medium, aiming to improve the technical problem of how to detect and identify items in the cockpit.

[0004] In a first aspect, embodiments of this application provide a method for detecting abandoned items, the method comprising: When a passenger exit signal is detected, cabin images are acquired from multiple cameras located within the cabin. The cockpit image is input into a pre-trained object detection model to determine a list of cockpit object categories, including object location information and category information; Based on pre-set rules for determining abandoned items, determine whether the list of cabin items includes abandoned items; When the aforementioned abandoned item is included, an abandoned item reminder message is output.

[0005] Secondly, embodiments of this application provide a method for training an item detection model, the method comprising: Based on the cockpit image dataset, the feature differences between foreground targets and background are learned through the target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and background. Based on the cockpit image dataset and the binary classification detection model, foreground targets are extracted from each cockpit image to obtain an item classification dataset. Based on the item classification dataset, the model is trained to obtain the item category detection model and the initial category template set; Remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors; Based on the feature extractor and the initial category template set, an item category updater is determined; The item detection model is determined based on the binary classification detection model, the item category detection model, and the item category updater.

[0006] Thirdly, embodiments of this application also provide a device for detecting abandoned items, the device comprising: The cockpit image acquisition module is used to acquire cockpit images from multiple cameras in the cockpit when a passenger exit signal is detected. The determination module is used to input the cockpit image into a pre-trained item detection model to determine a list of cockpit item categories; The judgment module is used to determine whether the cabin item category list includes abandoned items based on the pre-set abandoned item judgment rules; The reminder module is used to output a reminder message for the abandoned item when it is included.

[0007] Fourthly, embodiments of this application also provide an object detection model training device, the device comprising: The classification detection model determination module is used to learn the feature differences between foreground targets and background based on the cockpit image dataset and through the target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and background. The item classification dataset determination module is used to extract foreground targets from each cockpit image based on the cockpit image dataset and the binary classification detection model to obtain the item classification dataset; The training module is used to train the model based on the item classification dataset to obtain the item category detection model and the initial category template set; The feature extractor determination module is used to remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors; The item category updater determination module is used to determine the item category updater based on the feature extractor and the initial category template set; The item detection model determination module is used to determine the item detection model based on the binary classification detection model, the item category detection model, and the item category updater.

[0008] Fifthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to implement the abandoned item detection method described in the first aspect or the item detection model training method described in the second aspect.

[0009] Sixthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the abandoned item detection method described in the first aspect or the item detection model training method described in the second aspect.

[0010] The technical solution of this application embodiment acquires cabin images from multiple cameras in the cabin when a passenger exit signal is detected. These images are then input into a pre-trained item detection model to determine a list of cabin item categories containing item location and category information. Based on pre-set rules for determining whether items are left behind, the system judges whether any items are left behind and outputs a reminder message when such items are found. This forms a complete closed loop for lost item detection, achieving accurate identification and positioning of cabin items. Furthermore, the rules for determining lost items accurately filter out items requiring reminders, and the multi-channel reminder output ensures timely notification to users, effectively preventing items from being left behind, protecting user property safety, and simultaneously improving the intelligence level and user experience of the smart cabin. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the method for detecting abandoned items provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the item detection model training method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the abandoned items detection device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the item detection model training device provided in the embodiments of this application; Figure 5 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] With the rapid development of automotive intelligence and automation technologies, in-vehicle perception systems are playing an increasingly crucial role in smart cockpits. In recent years, in-vehicle cameras, infrared sensors, millimeter-wave radar, and other sensing devices have been widely used in mid-to-high-end models and intelligent connected vehicles. These are not only used for driver behavior monitoring, such as fatigue driving and attention detection, but also extensively applied to understanding the in-vehicle environment, such as passenger detection and cabin status recognition. At the same time, with the rise of shared mobility, ride-hailing, driverless taxis, and automated parking, the phenomenon of passengers leaving personal belongings in vehicles is becoming increasingly frequent. For example, it is very common for users to leave behind items such as mobile phones, wallets, water bottles, documents, and umbrellas after getting out of the car. This not only concerns the safety of users' property but also brings problems such as increased customer service costs, damage to brand image, and even legal liability to operators. Therefore, how to promptly detect and alert drivers to items left in the vehicle before they leave has become a critical issue that intelligent cockpit systems urgently need to address.

[0014] Based on this, the present application provides a method for detecting lost and found items. This method acquires cabin images from multiple cameras within the cabin when a passenger exit signal is detected. The images are then input into a pre-trained item detection model to determine a list of cabin item categories containing item location and category information. A pre-defined rule for determining lost and found items is then used to determine if any items are found. If so, a reminder message is output, forming a complete closed loop for detecting lost and found items. This achieves accurate identification and location of cabin items, precisely filters out items requiring reminders through the rule for determining lost and found items, and ensures timely notification to users through multi-channel reminder information output. This effectively prevents the loss of lost and found items, protects user property safety, and improves the intelligence level and user experience of the smart cabin.

[0015] This application provides a method for detecting abandoned items. Please refer to the following embodiments. Figure 1 This includes the following steps: S110 acquires cabin images from multiple cameras inside the cabin when it detects a passenger leaving the vehicle signal.

[0016] The abandoned item detection method provided in this application is applied to an abandoned item detection system, on which a pre-trained item detection model is installed.

[0017] Specifically, the abandoned items detection system can monitor passenger exit signals. When a passenger exit signal is detected, it immediately triggers multiple cameras to capture images of the cabin. These multiple cameras refer to image acquisition devices deployed in key areas such as the center console, roof, seat sides, and trunk. Their layout must achieve complete coverage of the cabin, ensuring that all areas where items may have been left behind, including seats, floor, center console, and trunk, are captured.

[0018] Specifically, the passenger exit signal can be a signal triggered when a preset signal is detected. Preset signals include vehicle ignition off, driver and passenger seatbelts unfastened, doors unlocked, and the user leaving the vehicle with the smart key. To avoid accidental triggering by temporary operations, an anti-shake mechanism can be set, such as triggering the signal only after the preset signal has been stable for a period of time.

[0019] S120, the cockpit image is input into a pre-trained item detection model to determine a list of cockpit item categories, including item location information and category information.

[0020] The item detection model is an integrated model of binary classification detection, item category detection, and item category updater, with functions of target localization, category recognition, and dynamic category expansion.

[0021] After acquiring cockpit images, they are input into a pre-trained object detection model. The model processes the images according to a localization, extraction, and recognition process. First, a binary classification detection model quickly segments the foreground and background, outlining the locations of all suspected objects and outputting their coordinates. Next, a feature extractor extracts high-dimensional feature vectors of the objects, performs similarity matching with a category template feature library, and combines this with an object category updater to determine the object category. Finally, all detected objects are organized according to their location and category information to form a cockpit object category list. The location information clearly indicates the specific area and coordinates of the object, while the category information indicates the category to which the object belongs. Objects that cannot be classified into a known category are marked as "unknown objects" in the list to ensure no object is missed due to category incompleteness and to provide data for subsequent category updates.

[0022] S130, based on the pre-set rules for determining abandoned items, determine whether the list of cabin items includes abandoned items.

[0023] The pre-defined rules for identifying lost and found items serve as the standard for distinguishing between items that require reminders and those that do not. These rules need to be customized to suit the characteristics of the in-vehicle environment and individual user needs, balancing versatility and flexibility. The rules should encompass at least three dimensions: exclusion of inherent items, user-defined settings, and scenario-based judgment. This ensures that items that truly require reminders are accurately identified while avoiding false reminders for items intentionally left by the user.

[0024] By verifying each item in the cabin item category list one by one and analyzing it against the rules for determining lost items, we exclude items that are inherent to the vehicle and user-defined items that do not require reminders. We then filter out items that belong to the key reminder category or unknown category. At the same time, we combine the frequency of the items and their location characteristics to determine whether they are temporarily placed items, and finally determine whether there are any lost items that require reminders.

[0025] For example, in the pre-defined rules for determining abandoned items, inherent items that do not require reminders include car air fresheners and car chargers, user-defined items that do not require reminders include a tissue box, and items requiring reminders are mobile phones, ID cards, and documents. The system verifies "Center console - mobile phone" and "Rear floor left side - documents" in the cabin item category list. Neither the mobile phone nor the documents belong to the inherent or user-defined categories that do not require reminders, and both belong to the categories requiring reminders. Furthermore, both items are present in multiple frames of images, and their positions are far from the car door, ruling out the possibility of temporary placement. Therefore, both items are determined to be abandoned items. If "Center console - car air freshener" appears in the list, it is directly excluded according to the inherent item rules and is not determined to be abandoned item.

[0026] S140, when the abandoned item is included, output abandoned item reminder information.

[0027] If the rules determine that there is leftover items, the user will be notified through multiple channels and with high priority to ensure that the user receives the reminder in a timely manner and can handle it. At the same time, user interaction and feedback are supported to form a closed loop.

[0028] The alert message should include at least the type and specific location of the lost item, so that users can quickly locate the item.

[0029] Specifically, standardized reminder content is generated based on the information of the items left behind, and the reminder is output simultaneously through multiple methods such as in-vehicle voice, central control screen pop-up, mobile APP push, and car key vibration. At the same time, a user feedback interface is reserved to support users to confirm the processing through voice, touch and other methods. After the processing is completed, the system automatically closes the reminder to avoid repeated reminders.

[0030] For example, upon determining the presence of two items left behind: "phone on the center console" and "documents on the left side of the rear floor," an alert message is immediately generated. First, the in-vehicle voice system broadcasts the alert at a volume higher than normal, repeating it twice to ensure the user hears it. A prominent red pop-up window appears on the center screen, displaying a diagram of the item's location and category, and providing "Confirmed" and "Ignore" buttons. Simultaneously, an alert message is pushed to the user's linked vehicle app, along with a screenshot of the cabin image indicating the item's location. If the user has already exited the vehicle and attempted to lock it with the smart key, the system does not lock the car immediately; the car key vibrates as feedback, and the voice broadcast is triggered again. After the user returns to the vehicle and retrieves the items, they can announce "retrieved" via voice. The system automatically re-checks the cabin image, confirms no items are left behind, deactivates all alerts, and allows normal locking.

[0031] This application embodiment acquires cabin images from multiple cameras within the cabin when a passenger exit signal is detected. These images are then input into a pre-trained item detection model to determine a list of cabin item categories containing item location and category information. Based on pre-set rules for identifying lost items, the system determines whether any items are left behind and outputs a reminder message when such items are found. This forms a complete closed loop for lost item detection, achieving accurate identification and location of cabin items. Furthermore, the rules for identifying lost items accurately filter out items requiring reminders, and the multi-channel reminder output ensures timely notification to users, effectively preventing the loss of items, protecting user property safety, and simultaneously improving the intelligence level and user experience of the smart cabin.

[0032] In an optional embodiment of this application, before detecting a passenger disembarkation signal, the method further includes: The system receives a pre-trained object detection model sent by the server; the object detection model is trained by the server based on a cockpit image dataset. The item detection model includes a binary classification detection model for distinguishing foreground objects from background, an item category detection model for determining item categories, and an item category updater for determining whether to add a new category template.

[0033] Before detecting passenger disembarkation signals, the system needs to receive a pre-trained object detection model sent by the server. This object detection model is trained by the server based on the cabin image dataset and includes a binary classification detection model for distinguishing foreground objects from background, an object category detection model for determining object categories, and an object category updater for determining whether to add a new category template.

[0034] The above-described implementation scheme of this application trains an item detection model, which includes a binary classification detection model, an item category detection model, and an item category updater, on the cockpit image dataset by the server. This model is then sent to the abandoned item detection system for use by the abandoned item detection method. This enables the abandoned item detection system to have the item detection function, providing high-performance and reliable technical support for the abandoned item detection function in intelligent cockpits.

[0035] In an optional embodiment of this application, the cockpit image is input into a pre-trained object detection model to determine a list of cockpit object categories, including object location information and category information, including: The object detection model includes a binary classification detection model to identify foreground targets in the cockpit image, thereby determining multiple foreground targets and the object location information corresponding to each foreground target. For each foreground target, the object detection model includes an object category detection model to perform category detection on the foreground target, thereby obtaining the category information corresponding to the foreground target; Based on the type information and item location information corresponding to the multiple foreground targets, the list of cabin item categories is determined.

[0036] In determining the cabin item category list, this application embodiment first uses a binary classification detection model to locate foreground targets and obtain their location information, and then uses an item category detection model to determine the category.

[0037] Specifically, the binary classification detection model is a sub-model within the object detection model, used to distinguish foreground objects from the background in cockpit images. The cockpit image obtained by the object detection model is input into the binary classification detection model. This model analyzes the cockpit image based on pre-learned feature differences between foreground objects and the background, identifying foreground objects. Simultaneously, a localization algorithm determines the specific coordinate range of each foreground object in the cockpit image, obtaining the object location information corresponding to each foreground object. At this point, a set of multiple foreground objects is obtained, along with the object location information corresponding to each foreground object.

[0038] Furthermore, the item category detection model is another sub-module in the item detection model. For the foreground target output by the binary classification model, the item category detection model can extract the high-dimensional feature vector of the foreground target through the built-in feature extractor, and compare the similarity of this vector with the initial category template feature library in the model. The initial category template feature library stores the standard feature vectors of various known items. Based on the similarity comparison result and the preset first threshold, it can be determined whether the foreground target belongs to a known category or an unknown category, thereby obtaining the category information corresponding to the foreground target.

[0039] After obtaining the category information of each foreground target, all valid items are recorded and organized to obtain a cockpit item category list, which includes the category information and location information of each detected item.

[0040] The above-described implementation scheme of this application uses a binary classification detection model in the object detection model to identify foreground targets in the cockpit image, determine multiple foreground targets and their corresponding object location information, then uses an object category detection model to perform category detection on each foreground target to obtain the corresponding category information, and finally integrates the category information of the foreground targets with the object location information to determine the cockpit object category list, thereby realizing the acquisition of the location and category of objects in the cockpit and forming a structured object information list, providing reliable data support for subsequent identification of abandoned objects.

[0041] In an optional embodiment of this application, the foreground target is classified using an item category detection model included in the item detection model to obtain category information corresponding to the foreground target, including: The foreground target is feature extracted using the item category detection model to obtain the feature vector corresponding to the foreground target; Calculate the similarity between the feature vector and each category template in the initial category template set; The calculated similarity is compared with a preset first threshold to determine whether the foreground target belongs to a known category. When the foreground target belongs to a known category, the matched category is determined as the category information corresponding to the foreground target; When the foreground target does not belong to a known category, the unknown category is determined as the category information corresponding to the foreground target.

[0042] In this embodiment of the application, when determining the category information of the foreground target, the features of the foreground target are extracted by a category detection model and converted into a quantifiable feature vector. Then, the category labeling of the foreground target is achieved by similarity comparison and threshold judgment.

[0043] Specifically, the item category detection model includes a feature extractor, which extracts features from the foreground target to obtain a feature vector. This feature vector is a digital representation of the core features of the foreground target and is unique. Feature vectors of items of the same category are highly similar, while feature vectors of items of different categories are significantly different.

[0044] Furthermore, the similarity between the feature vector and each category template in the initial category template set is calculated. The initial category template set is the set of templates selected for each known category after the item category detection model has been trained. Each category template corresponds to a feature vector, which constitutes the initial category template feature library.

[0045] After obtaining the similarity results, they are compared one by one with a preset first threshold. The first threshold is used to distinguish whether the foreground target belongs to a known category or an unknown category. If the highest similarity value is greater than or equal to the first threshold, it indicates that the features of the foreground target highly match the features of the known category template, and the foreground target is determined to belong to the known category. If the similarity values ​​of all categories are less than the first threshold, it indicates that the features of the foreground target are significantly different from the features of all known category templates and cannot be classified into any known category. The foreground target is determined not to belong to the known category, that is, to belong to the unknown category.

[0046] Finally, when a foreground target is determined to belong to a known category, the standard name of the matching category is used as the category information corresponding to the foreground target. When a foreground target is determined not to belong to a known category, it indicates that its features are significantly different from the features of all categories in the initial category template set and cannot be classified into any known category. In this case, the "unknown category" is used as the category information corresponding to the foreground target.

[0047] The above-described implementation scheme of this application extracts features from the foreground target using an item category detection model to obtain a feature vector, calculates the similarity between the feature vector and each category template in the initial category template set, and determines whether the foreground target belongs to a known category based on a first threshold. This achieves the detection of the foreground target category, ensuring the accuracy of known category identification while also being compatible with unknown categories.

[0048] This application provides a method for training an item detection model. Please refer to the following embodiments. Figure 2 This includes the following steps: S210: Based on the cockpit image dataset, the feature differences between foreground targets and background are learned through a target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and background.

[0049] In this embodiment, the cockpit image dataset serves as the foundation for model training. It is obtained by capturing images of the cockpit from multiple locations within the cockpits of multiple vehicles under various preset scenarios. Each cockpit image in the dataset is labeled with a foreground object. These preset scenarios need to cover a variety of conditions, including urban roads, parking lots, day / night cycles, and different lighting and weather conditions. They also need to include key stages such as before the driver gets into the vehicle, during driving, and before the vehicle is turned off and the driver is about to leave. In a specific embodiment, approximately 200,000 original image frames are collected, covering all the aforementioned scenarios and stages.

[0050] After obtaining the cockpit image dataset, a model is trained based on the dataset to obtain a binary classification detection model for distinguishing foreground objects from background. During model training, a lightweight object detection architecture can be used, with "foreground / background" as the binary classification target. The labeled cockpit image dataset is input, and the model parameters are optimized through backpropagation until the model's accuracy on the validation set stabilizes, thus obtaining the binary classification detection model.

[0051] S220, Based on the cockpit image dataset and the binary classification detection model, extract the foreground targets in each cockpit image to obtain the item classification dataset.

[0052] The cockpit image dataset from S210 is input into a trained binary classification detection model, which outputs the precise location and confidence score of all foreground targets in each frame. First, invalid targets are filtered out using preset rules, such as retaining targets with a confidence score ≥ 0.7, removing targets that are too small or too large, and filtering targets that exceed the physical space of the cockpit. Then, based on the coordinates of the valid bounding boxes, image regions containing only objects are precisely cropped from the original cockpit images. These pure object images are then manually classified and labeled according to a preset category system, ultimately forming an object classification dataset.

[0053] S230, Based on the item classification dataset, train the model to obtain the item category detection model and the initial category template set.

[0054] After obtaining the item classification dataset, a lightweight architecture suitable for image classification is selected. Using the item classification dataset as training data, the training objective is set as "determining which category a given pure item image belongs to in a preset category." During training, the model learns the feature differences between different item categories through a multi-layer network until the classification accuracy on the validation set stabilizes, thus obtaining the item category detection model. Simultaneously, representative template images are selected from the item classification dataset for each category, covering different poses, lighting conditions, and slightly occluded scenes, forming an initial category template set. For example, 100 template images are selected for the mobile phone category, covering scenes such as front-facing, side-facing, and low-light environments.

[0055] S240, Remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors.

[0056] The network structure of an item category detection model includes a feature extraction layer and a classification head. The feature extraction layer transforms the input item image into a high-dimensional vector representing the item's core features. The classification head then outputs the probability of the item belonging to each category based on this high-dimensional vector. After training, the classification head is removed, and the feature extraction layer is retained. Typically, a block output layer or a fully connected layer at the back end of the network is chosen, satisfying the principle of high inter-class discrimination and low intra-class discrimination, resulting in the feature extractor. This feature extractor transforms the input item image into a standardized high-dimensional feature vector.

[0057] S250, Based on the feature extractor and the initial category template set, determine the item category updater.

[0058] After obtaining the feature extractor and the initial category template set, the item category updater for dynamic category expansion is determined. First, the feature extractor extracts features from each template image in the initial category template set, obtaining a high-dimensional feature vector for each template. This vector is then stored according to category (e.g., in .npy format) to form the initial category template feature library. Further, cosine similarity is selected as the target similarity calculation formula, a first threshold is set to determine whether an item belongs to a known category, and a new category addition strategy is defined. Finally, the aforementioned template feature library, similarity calculation formula, threshold, and addition strategy are integrated to form the item category updater.

[0059] S260, determine the item detection model based on the binary classification detection model, the item category detection model, and the item category updater.

[0060] The three core components are integrated and encapsulated according to localization, recognition, and logic to form a complete object detection model. The input of this object detection model is a real-time cockpit image. First, a binary classification detection model is used to locate and crop the foreground target. Then, the pure object image is input into the object category detection model for preliminary classification. At the same time, a feature extractor extracts feature vectors and performs similarity matching with the template feature library in the object category updater to accurately determine the object category. If the category is unknown, the addition strategy of the object category updater is triggered. Finally, the detection results of the object location and object category are output.

[0061] In this embodiment, a binary classification detection model, an item category detection model, and an item category updater are trained using a cockpit image dataset to obtain an item detection model. This model enables accurate location and category recognition of items within the cockpit. The dynamic category expansion capability breaks through the limitations of fixed categories, improving the generalization and flexibility of item detection. It supports personalized category management and continuous optimization without the need for additional hardware installation. This provides reliable technical support for the detection and alerting of abandoned items in smart cockpit scenarios, comprehensively improving detection performance and user experience.

[0062] In an optional embodiment of this application, determining an item category updater based on the feature extractor and the initial category template set includes: The feature extractor extracts features from each template in the initial category template set to obtain an initial category template feature library, which stores a high-dimensional feature vector corresponding to each initial category template. The following are defined: a target similarity calculation formula for calculating the similarity between the item to be confirmed and the initial category template feature library; a first threshold for determining whether the item to be confirmed belongs to the initial category template set; and a new strategy for determining whether to add a new category template. The item category updater is determined based on the target similarity calculation formula, the first threshold, and the addition strategy.

[0063] In this embodiment, the initial category template set is a set of representative template images selected for each preset category after the item category detection model has been trained. These templates need to cover different poses, lighting conditions, and slightly occluded scenes of the corresponding item category to ensure that they can reflect the core features of the item category. The feature extractor is the feature extraction layer retained after removing the classification head of the item category detection model, and it has the ability to convert any item image into a high-dimensional feature vector.

[0064] When determining the item category updater, firstly, each template image in the initial category template set is input into the feature extractor. Through multi-layer network operations in the feature extraction layer, the physical features of each template are transformed into standardized high-dimensional feature vectors. Then, these vectors are classified and organized according to category to form the initial category template feature library. This feature library is a mapping set between categories and high-dimensional feature vectors, storing the digital representation of the features of each category of items, providing a standard reference for subsequent similarity matching.

[0065] Furthermore, it clarifies the three key logics of how to determine similarity, how to identify known categories, and how to add unknown categories, ensuring that the category update process is standardized, accurate, and meets actual needs.

[0066] Among these considerations, determining the target similarity calculation formula requires selecting one that is suitable for high-dimensional feature vector comparison, has low computational cost, and strong discriminative power, prioritizing compatibility with the computing power limitations of in-vehicle terminals. The similarity calculation of high-dimensional vectors should emphasize directional consistency and feature correlation between vectors, avoiding interference from the absolute numerical magnitude of the vectors.

[0067] When setting the first threshold, it is the core criterion for determining whether an item to be confirmed belongs to the initial category template set. A balance must be struck between "precision" and "recall." A threshold that is too high may lead to some similar items being misclassified as unknown categories, while a threshold that is too low may lead to items from different categories being misclassified as the same category. The threshold setting should be based on the vector distribution characteristics of the initial category template feature library, combined with the accuracy requirements of the actual detection scenario, to ensure effective differentiation between known and unknown categories.

[0068] When formulating new strategies, it is necessary to clarify the logic of unknown category determination, unknown category storage, and new category triggering conditions. When the similarity between the item to be confirmed and all categories in the initial category template feature library is lower than the first threshold, it is determined to be an unknown category and its feature vector is stored. When the same type of unknown item appears multiple times and the feature similarity meets the requirements, the category addition process is automatically triggered to convert the unknown item into a new known category and add it to the initial category template set.

[0069] For example, cosine similarity is chosen as the formula for calculating target similarity. This formula quantifies the similarity between the feature vector of the item to be identified and the template feature vector by calculating the cosine value of the angle between them; the closer the value is to 1, the higher the similarity. A first threshold of 0.8 is set, meaning that if the cosine similarity between the vector of the item to be identified and the template vector of a certain category is ≥0.8, it is determined to belong to that category; if all are <0.8, it is determined to be an unknown category. A new strategy is set to store the feature vectors of unknown category items in an unknown category library. When the similarity between a vector of an unknown item and multiple vectors in the unknown category library is ≥0.8, and the number of vectors meeting this condition reaches a preset requirement, the category is automatically added, and these vectors are used as the initial template vectors for the new category to supplement the initial category template feature library. For example, if the item to be confirmed is "tablet computer", its feature vector has a similarity of 0.6 (<0.8) with the template vector of "mobile phone", and its similarity with other categories is also less than 0.8. It is determined to be an unknown category and stored in the unknown category library. If "tablet computer" is detected multiple times in the future, and its vector has a similarity of ≥0.8 with the "tablet computer" vector already stored in the unknown category library, and the cumulative number of detections reaches 5 times, the "tablet computer" category will be automatically added.

[0070] Finally, the constructed initial category template feature library, similarity calculation formula, first threshold, and addition strategy are integrated and encapsulated to form a complete item category updater. During integration, the linkage logic of each component must be clearly defined. After confirming the input of the item's feature vector, the similarity is first compared one by one with all category vectors in the initial category template feature library using the target similarity calculation formula to obtain the similarity result. Then, based on the first threshold, it is determined whether the item is a known or unknown category. If it is a known category, the corresponding category is directly output; if it is an unknown category, it is stored in the unknown category library according to the addition strategy, and it is determined whether the conditions for adding a new category are met. If the conditions for adding a new category are met, a new category is automatically created, and the relevant vectors are added to the initial category template feature library, completing the category expansion. Simultaneously, data transmission interfaces between components need to be designed to ensure smooth flow of data such as feature vectors, similarity results, and category determination information. Exception handling logic also needs to be added to handle special cases such as feature library read failures and similarity calculation conflicts, ensuring the stable operation of the item category updater.

[0071] The above-described implementation scheme of this application constructs an initial category template feature library by extracting features from the initial category template set using a feature extractor. It then integrates the target similarity calculation formula, the first threshold, and the addition strategy to form an item category updater, achieving accurate matching of known categories. Furthermore, the addition strategy breaks through the fixed category limitation to achieve dynamic expansion of unknown categories, balancing the consistency and generalization of detection. It adapts to the lightweight and real-time requirements of in-vehicle scenarios, providing flexible and reliable technical support for intelligent cockpit item detection and effectively improving detection results and user experience.

[0072] In an optional embodiment of this application, determining the addition strategy for determining whether to add a new category template includes: When the similarity between the item to be confirmed and each initial template in the initial category template set is less than the first threshold, the category of the item to be confirmed is confirmed as an unknown category and added to the unknown category library; Calculate the similarity between the item to be confirmed and each item in the unknown category library, and determine the number of items with a similarity greater than a second threshold; When the number of items exceeds the preset number, a new category is determined and added to the initial category template set.

[0073] In this embodiment of the application, the items to be confirmed refer to cabin items whose category needs to be determined after being located and feature extracted by the item detection model. The first threshold is a pre-set standard used to distinguish whether an item belongs to a known category.

[0074] Specifically, the similarity is first calculated using the target similarity calculation formula, comparing the high-dimensional feature vector of the item to be classified with the feature vectors of all initial templates in the initial category template set. If all similarity results are below the first threshold, it indicates that the item's features differ significantly from those of existing known categories and cannot be classified into any defined category; therefore, it is classified as an unknown category. Simultaneously, the high-dimensional feature vector of this unknown category, the corresponding item image, and the detection time are stored in an unknown category library to provide data for subsequent similarity analysis and category addition determination.

[0075] Furthermore, when a new item to be confirmed is determined to be in an unknown category and is prepared to be stored in the unknown category library, a similarity statistical analysis must first be performed to determine whether the item belongs to the same undefined potential new category. Using the target similarity calculation formula, the high-dimensional feature vector of the item to be confirmed is compared with the feature vectors of all items already stored in the unknown category library to calculate their similarity. Simultaneously, a second threshold is set as the standard for judging "same type of item". If the similarity is greater than the second threshold, it indicates that their core features are highly consistent and they belong to the same category of item. The number of all items that meet the similarity requirement of greater than the second threshold is counted. This number directly reflects the frequency and stability of the potential new category of item, and is a key basis for subsequent determination of whether to add a new category.

[0076] The preset quantity is set based on the occurrence pattern of items in the cockpit scene. It is the minimum frequency of occurrence of the same type of item that triggers the addition of a new category and serves as a standard for judging whether a potential new category has the value of continuous appearance.

[0077] The number of items with a similarity greater than the second threshold is compared with a preset number. If the number exceeds the preset number, it indicates that the potential new category of items continues to appear in the cockpit scene and has the value of being included in the known categories; therefore, the new category is determined to be added. When adding a new category, it needs to be named. It can be named "Unknown Category-XX" by default, and users can modify it later. All feature vectors and item images belonging to this category in the unknown category library are used as initial templates to form the template set of the new category. Finally, the new category and its corresponding template set are added to the initial category template set. At the same time, the initial category template feature library needs to be updated synchronously to include the template feature vector of the new category, so that items of this type can be directly identified as known categories during subsequent detection.

[0078] The above-described implementation scheme of this application determines items of the unconfirmed category that cannot be attributed to the initial category template set as unknown categories and adds them to the unknown category library by setting a first threshold. It calculates the similarity between the items to be confirmed and the items in the unknown category library by combining a second threshold and counts the number of items that meet the criteria. When the number of items meets the condition, a new category is determined and added to the initial category template set, forming a category template addition strategy. This realizes the dynamic expansion of categories to break through the limitations of fixed categories. The accuracy of the new categories is ensured by double threshold verification, avoiding the generation of redundant categories and improving the generalization and flexibility of the item detection system. This provides reliable support for the continuous optimization of the item detection function in the intelligent cockpit.

[0079] In an optional embodiment of this application, before training a model based on a cockpit image dataset to obtain a binary classification detection model for distinguishing foreground targets and background, the method includes: Under multiple preset scenarios, the cockpit is photographed by cameras placed inside the cockpit to obtain an initial cockpit image dataset; The cockpit image dataset is obtained by segmenting and labeling the foreground targets of each cockpit image in the initial cockpit image dataset using a segmentation model. The cockpit image dataset includes cockpit images corresponding to multiple preset scenes, and each cockpit image is labeled with a foreground target.

[0080] In this embodiment of the application, when acquiring the cockpit image dataset used to determine the object detection model, multiple preset scenarios are pre-defined. These preset scenarios are based on the actual usage needs of the smart cockpit, covering a set of cockpit scenes with different spatial environments, weather conditions, lighting conditions, and riding stages, ensuring that the acquired images can cover all training scenarios required by the binary classification detection model. The cameras deployed inside the cockpit refer to image acquisition devices installed at multiple key locations within the cockpit, which must achieve seamless coverage of all areas inside the cockpit where objects may be placed.

[0081] According to the pre-defined scenario plan, the cameras inside the cockpit continuously capture images of the cockpit interior in each scenario. During the capture process, the camera parameters must remain consistent, and the scene information corresponding to each image must be recorded. The raw images captured in all scenarios are then aggregated, and invalid images due to equipment malfunction or shooting errors are removed to form an initial cockpit image dataset.

[0082] Furthermore, a segmentation model is determined. This model is a deep learning model with pixel-level image segmentation capabilities, which can accurately identify the category of each pixel in the image and achieve separation of foreground objects from the background. Foreground objects refer to various items inside the cabin, while the background refers to the inherent interior structure of the cabin.

[0083] Each image in the initial cockpit image dataset is input into the segmentation model. The model, through its learned feature extraction and classification capabilities, performs pixel-level processing to identify and segment all foreground target regions, generating corresponding annotations. These annotations must include the location contours and category identifiers of the foreground targets, establishing a one-to-one correspondence with the original images. All segmented and annotated cockpit images are then integrated with their corresponding annotations to form the final cockpit image dataset. This dataset serves as the direct data source for training the subsequent binary classification detection model.

[0084] The above-described implementation scheme of this application obtains an initial cockpit image dataset by capturing images in multiple preset scenarios using in-cabin cameras. Then, a segmentation model is used to segment and label the foreground targets in each cockpit image to obtain the cockpit image dataset. This ensures the scene diversity and labeling completeness of the dataset, providing a high-quality and highly available data source for the training of subsequent binary classification detection models. Automated labeling improves data processing efficiency, reduces labeling costs, and enhances the reusability of the dataset. This supports the subsequent model in improving generalization ability and detection accuracy, adapting to the actual application needs of intelligent cockpits.

[0085] This application also provides a device for detecting abandoned items; please refer to [link / reference]. Figure 3 The abandoned item detection device 30 includes: The cockpit image acquisition module 310 is used to acquire cockpit images obtained by multiple cameras in the cockpit when a passenger exit signal is detected. The determination module 320 is used to input the cockpit image into a pre-trained item detection model to determine a list of cockpit item categories; The judgment module 330 is used to determine whether the cabin item category list includes abandoned items based on the pre-set abandoned item judgment rules; The reminder module 340 is used to output a reminder message for the abandoned item when the abandoned item is included.

[0086] Optionally, before detecting a passenger leaving the vehicle signal, the device further includes: The receiving module is used to receive a pre-trained object detection model sent by the server; the object detection model is obtained by the server through model training based on the cockpit image dataset; the object detection model includes a binary classification detection model for distinguishing foreground objects and background, an object category detection model for determining object categories, and an object category updater for determining whether to add a new category template.

[0087] Optionally, the determination module includes: The first determining submodule is used to perform foreground target recognition on the cockpit image through the binary classification detection model included in the item detection model, and to determine multiple foreground targets and the item location information corresponding to each foreground target; The detection submodule is used to perform category detection on each foreground target using the item category detection model included in the item detection model, and obtain the category information corresponding to the foreground target. The second determining submodule is used to determine the cabin item category list based on the type information and item location information corresponding to the multiple foreground targets.

[0088] Optionally, the detection submodule includes: The feature extraction unit is used to extract features from the foreground target using the item category detection model to obtain the feature vector corresponding to the foreground target; A calculation unit is used to calculate the similarity between the feature vector and each category template in the initial category template set; The first determining unit is used to determine whether the foreground target belongs to a known category by comparing the calculated similarity with a preset first threshold. The second determining unit is used to determine the matched category as the category information corresponding to the foreground target when the foreground target belongs to a known category; The third determining unit is used to determine the unknown category as the category information corresponding to the foreground target when the foreground target does not belong to the known category.

[0089] This application also provides an object detection model training device. Please refer to [link / reference]. Figure 4 The item detection model training device 40 includes: The classification detection model determination module 410 is used to learn the feature differences between foreground targets and backgrounds based on the cockpit image dataset and through the target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and backgrounds. The item classification dataset determination module 420 is used to extract foreground targets in each cockpit image based on the cockpit image dataset and the binary classification detection model to obtain the item classification dataset; The training module 430 is used to train the model based on the item classification dataset to obtain the item category detection model and the initial category template set; The feature extractor determination module 440 is used to remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors; The item category updater determination module 450 is used to determine the item category updater based on the feature extractor and the initial category template set; The item detection model determination module 460 is used to determine the item detection model based on the binary classification detection model, the item category detection model, and the item category updater.

[0090] Optional, the item category updater determination module includes: The feature extraction submodule is used to extract features from each template in the initial category template set through the feature extractor to obtain an initial category template feature library, wherein the initial category template feature library stores a high-dimensional feature vector corresponding to each initial category template. The third determination submodule is used to determine the target similarity calculation formula for calculating the similarity between the item to be confirmed and the initial category template feature library, the first threshold for determining whether the item to be confirmed belongs to the initial category template set, and the addition strategy for determining whether to add a new category template. The fourth determination submodule is used to determine the item category updater based on the target similarity calculation formula, the first threshold, and the addition strategy.

[0091] Optionally, the third determination submodule includes: The fourth determining unit is used to determine the category of the item to be confirmed as an unknown category and add it to the unknown category library when the similarity between the item to be confirmed and each initial template in the initial category template set is less than the first threshold. The fifth determining unit is used to calculate the similarity between the item to be confirmed and each item in the unknown category library, and to determine the number of items with a similarity greater than the second threshold. The sixth determining unit is used to determine a new category when the number of items is greater than a preset number, and to add the new category to the initial category template set.

[0092] Optionally, before training a model based on the cockpit image dataset to obtain a binary classification detection model for distinguishing foreground objects and background, the apparatus includes: The acquisition module is used to capture images of the cockpit using cameras placed inside the cockpit under multiple preset scenarios to obtain an initial cockpit image dataset; The processing module is used to segment and label the foreground targets of each cockpit image in the initial cockpit image dataset using a segmentation model, so as to obtain the cockpit image dataset. The cockpit image dataset includes cockpit images corresponding to multiple preset scenes, and each cockpit image is labeled with a foreground target.

[0093] This application also provides an electronic device, please refer to... Figure 5 It includes a processor 510 and a memory 520, wherein the memory 510 is used to store computer programs; the processor 520 is used to execute the programs stored in the memory 510 to implement the item detection model training method or the abandoned item detection method described in any embodiment of this application.

[0094] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the item detection model training method or the abandoned item detection method described in any embodiment of this application.

[0095] In this application, "multiple" refers to two or more.

[0096] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0097] The terms “first,” “second,” “third,” “fourth,” etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0098] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0099] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if the method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if the method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0100] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting abandoned items, characterized in that, The method includes: When a passenger exit signal is detected, cabin images are acquired from multiple cameras located within the cabin. The cockpit image is input into a pre-trained object detection model to determine a list of cockpit object categories, including object location information and category information; Based on pre-set rules for determining abandoned items, determine whether the list of cabin items includes abandoned items; When the aforementioned abandoned item is included, an abandoned item reminder message is output.

2. The method for detecting abandoned items according to claim 1, characterized in that, Before detecting a passenger disembarkation signal, the method further includes: The system receives a pre-trained object detection model sent by the server; the object detection model is trained by the server based on a cockpit image dataset. The item detection model includes a binary classification detection model for distinguishing foreground objects from background, an item category detection model for determining item categories, and an item category updater for determining whether to add a new category template.

3. The method for detecting abandoned items according to claim 1, characterized in that, The cockpit image is input into a pre-trained object detection model to determine a list of cockpit object categories, including object location information and category information, including: The object detection model includes a binary classification detection model to identify foreground targets in the cockpit image, thereby determining multiple foreground targets and the object location information corresponding to each foreground target. For each foreground target, the object detection model includes an object category detection model to perform category detection on the foreground target, thereby obtaining the category information corresponding to the foreground target; Based on the type information and item location information corresponding to the multiple foreground targets, the list of cabin item categories is determined.

4. The method for detecting abandoned items according to claim 3, characterized in that, The foreground target is classified using the item category detection model included in the item detection model to obtain the category information corresponding to the foreground target, including: The foreground target is feature extracted using the item category detection model to obtain the feature vector corresponding to the foreground target; Calculate the similarity between the feature vector and each category template in the initial category template set; The calculated similarity is compared with a preset first threshold to determine whether the foreground target belongs to a known category. When the foreground target belongs to a known category, the matched category is determined as the category information corresponding to the foreground target; When the foreground target does not belong to a known category, the unknown category is determined as the category information corresponding to the foreground target.

5. A method for training an object detection model, characterized in that, The method includes: Based on the cockpit image dataset, the feature differences between foreground targets and background are learned through the target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and background. Based on the cockpit image dataset and the binary classification detection model, foreground targets are extracted from each cockpit image to obtain an item classification dataset. Based on the item classification dataset, the model is trained to obtain the item category detection model and the initial category template set; Remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors; Based on the feature extractor and the initial category template set, an item category updater is determined; The item detection model is determined based on the binary classification detection model, the item category detection model, and the item category updater.

6. The method for training an item detection model according to claim 5, characterized in that, Based on the feature extractor and the initial category template set, an item category updater is determined, including: The feature extractor extracts features from each template in the initial category template set to obtain an initial category template feature library, which stores a high-dimensional feature vector corresponding to each initial category template. The following are defined: a target similarity calculation formula for calculating the similarity between the item to be confirmed and the initial category template feature library; a first threshold for determining whether the item to be confirmed belongs to the initial category template set; and a new strategy for determining whether to add a new category template. The item category updater is determined based on the target similarity calculation formula, the first threshold, and the addition strategy.

7. The method for training an item detection model according to claim 6, characterized in that, Determine the addition strategy used to determine whether to add a new category template, including: When the similarity between the item to be confirmed and each initial template in the initial category template set is less than the first threshold, the category of the item to be confirmed is confirmed as an unknown category and added to the unknown category library; Calculate the similarity between the item to be confirmed and each item in the unknown category library, and determine the number of items with a similarity greater than a second threshold; When the number of items exceeds the preset number, a new category is determined and added to the initial category template set.

8. The method for training an item detection model according to claim 5, characterized in that, Before obtaining a binary classification detection model to distinguish foreground targets and background by learning the feature differences between foreground targets and background using an object detection algorithm based on a cockpit image dataset, the method includes: Under multiple preset scenarios, the cockpit is photographed by cameras placed inside the cockpit to obtain an initial cockpit image dataset; The cockpit image dataset is obtained by segmenting and labeling the foreground targets of each cockpit image in the initial cockpit image dataset using a segmentation model. The cockpit image dataset includes cockpit images corresponding to multiple preset scenes, and each cockpit image is labeled with a foreground target.

9. A device for detecting abandoned items, characterized in that, include: The cockpit image acquisition module is used to acquire cockpit images from multiple cameras in the cockpit when a passenger exit signal is detected. The determination module is used to input the cockpit image into a pre-trained item detection model to determine a list of cockpit item categories; The judgment module is used to determine whether the cabin item category list includes abandoned items based on the pre-set abandoned item judgment rules; The reminder module is used to output a reminder message for the abandoned item when it is included.

10. An object detection model training device, characterized in that, include: The classification detection model determination module is used to learn the feature differences between foreground targets and background based on the cockpit image dataset and through the target detection algorithm to obtain a binary classification detection model for distinguishing foreground targets and background. The item classification dataset determination module is used to extract foreground targets from each cockpit image based on the cockpit image dataset and the binary classification detection model to obtain the item classification dataset; The training module is used to train the model based on the item classification dataset to obtain the item category detection model and the initial category template set; The feature extractor determination module is used to remove the classification head of the item category detection model to obtain a feature extractor for extracting item feature vectors; The item category updater determination module is used to determine the item category updater based on the feature extractor and the initial category template set; The item detection model determination module is used to determine the item detection model based on the binary classification detection model, the item category detection model, and the item category updater.

11. An electronic device, characterized in that, Including processor and memory; Among them, the memory is used to store computer programs; A processor is configured to execute a program stored in memory to implement the abandoned item detection method as described in any one of claims 1 to 4 or the item detection model training method as described in any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the abandoned item detection method as described in any one of claims 1 to 4 or the item detection model training method as described in any one of claims 5 to 8.

Images