A method for detecting items inside a vehicle and the vehicle

Abstract

This invention relates to the field of vehicle technology and discloses a method for detecting items inside a vehicle and a vehicle. The method includes: receiving a query request sent by a client, the query request being used to query for a target item; responding to the query request by calling an in-vehicle camera to capture an image of the vehicle interior, and identifying items in open areas of the vehicle interior through the in-vehicle image to obtain a first item detection result; responding to the query request by acquiring item attribute information in hidden areas, and inferring a second item detection result through the item attribute information, the item attribute information being obtained by scanning the hidden area using a sensor; and sending the first item detection result and the second item detection result to the client. Applying the technical solution of this invention can improve the accuracy of remotely detecting items inside a vehicle.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, specifically to a method for detecting items inside a vehicle and a vehicle. Background Technology

[0002] It's a common problem for users to inadvertently leave items in their cars when getting out, and it's also common for users to not remember whether they actually left anything behind. If the parking location is far from the user's current location, walking into the car to check can be time-consuming and disrupt their travel plans. Therefore, some technologies have proposed using mobile apps to remotely check items in cars. However, these technologies primarily rely on image recognition, so areas not covered by the car's cameras are often missed, making it impossible to determine if items were left behind. Users still need to walk into the car to check, wasting their time. Therefore, improving the accuracy of in-car item detection is a problem worth solving. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a method and vehicle for detecting items inside a vehicle, which solves the problem of inaccurate detection of items inside a vehicle in the prior art.

[0004] According to one aspect of the present invention, a method for detecting objects inside a vehicle is provided, applied to a vehicle. The method includes: receiving a query request sent by a client, the query request being used to query for a target object; responding to the query request by calling an in-vehicle camera to capture an image inside the vehicle, and identifying objects in open areas inside the vehicle using the in-vehicle image to obtain a first object detection result; responding to the query request by obtaining object attribute information in hidden areas, and inferring a second object detection result using the object attribute information, the object attribute information being obtained by scanning with a hidden area sensor; and sending the first object detection result and the second object detection result to the client.

[0005] According to another aspect of the present invention, an in-vehicle item detection device is provided, applied to a vehicle. The device includes: a request receiving module, configured to receive a query request sent by a client, the query request being for querying a target item; an open area detection module, configured to respond to the query request by calling an in-vehicle camera to capture an image of the vehicle interior, and identifying items in open areas of the vehicle interior through the in-vehicle image to obtain a first item detection result; a hidden area detection module, configured to respond to the query request by obtaining item attribute information in hidden areas, and inferring a second item detection result through the item attribute information, the item attribute information being obtained by scanning a hidden area sensor; and a detection result feedback module, configured to send the first item detection result and the second item detection result to the client.

[0006] According to another aspect of the present invention, a vehicle is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform the operation of the first aspect of the in-vehicle item detection method.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes a vehicle / device to perform the operation of the first aspect of the in-vehicle item detection method.

[0008] The technical solution provided by this invention has the following advantages:

[0009] This invention effectively solves the problem of missed detection in hidden areas caused by relying solely on image recognition in existing technologies by performing detection separately in open and hidden areas within the vehicle. For open areas, images are captured by the in-vehicle camera and items are identified, providing quick and intuitive item detection results. For hidden areas, the hidden area sensor scans to obtain item attribute information and infers the detection results, filling the detection gaps in areas not covered by the camera and achieving full coverage of in-vehicle item detection. Simultaneously, by receiving client query requests and feeding back the fused first and second item detection results, this method allows users to remotely confirm whether a target item is left in the vehicle without having to travel to or from the vehicle. This significantly reduces the time cost for users in confirming items and substantially improves the accuracy, comprehensiveness, and user convenience of in-vehicle item detection, effectively addressing the shortcomings of existing remote item query technologies, such as numerous detection blind spots and poor user experience.

[0010] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0011] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0012] Figure 1 This diagram illustrates a first embodiment of a method for detecting items inside a vehicle provided by the present invention.

[0013] Figure 2 This diagram illustrates a second embodiment of a method for detecting items inside a vehicle provided by the present invention.

[0014] Figure 3 A flowchart illustrating a third embodiment of a method for detecting items inside a vehicle provided by the present invention is shown.

[0015] Figure 4 This diagram illustrates the structure of an in-vehicle item detection device provided by the present invention.

[0016] Figure 5 A schematic diagram of an embodiment of a vehicle provided by the present invention is shown. Detailed Implementation

[0017] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0018] Figure 1 A flowchart of a first embodiment of a method for detecting items inside a vehicle according to the present invention is shown, the method being performed by a vehicle. Figure 1 As shown, the method includes the following steps:

[0019] Step S101: Receive a query request sent by the client. The query request is used to query the target item.

[0020] Step S102: In response to the query request, the in-vehicle camera is called to capture images inside the vehicle, and the items in the open areas inside the vehicle are identified through the in-vehicle images to obtain the first item detection result;

[0021] Step S103: In response to the query request, obtain the item attribute information of the hidden area, and infer the second item detection result through the item attribute information. The item attribute information is obtained by scanning the hidden area sensor.

[0022] Step S104: Send the detection results of the first item and the second item to the client.

[0023] Specifically, in this embodiment of the invention, when executing the in-vehicle item detection method, the vehicle's communication module receives a query request sent by a client (such as a user's mobile phone or tablet's in-vehicle control APP). This query request contains key identification information of the target item (such as the item name "black leather driver's license" and the item type "document type"), so that the system can clearly identify the detection object. For example, if a user finds that their driver's license is not with them when they are about to leave, they can send a query to the vehicle via the mobile APP to check "whether the black leather driver's license is in the vehicle". After receiving the request, the vehicle enters the subsequent detection process.

[0024] Subsequently, the vehicle's control module responds to the query request by calling an in-vehicle camera (e.g., a 1080P monocular infrared camera with 850nm infrared illumination, capable of clear imaging at night or in low-light conditions, avoiding blind spots) pre-installed on the roof or center console to capture images of the vehicle interior. Here, the open area inside the vehicle refers to areas that can be directly observed without mechanical operation, such as the trunk, seat surfaces, and center console. The in-vehicle images are then processed using an object detection model (e.g., the YOLOv8 object detection model, used as an example only, not a limitation). Image features are extracted (including but not limited to object outlines and textures; for example, if the target object is a suitcase, the outline features of its handle and wheels, as well as the texture features of its surface, can be extracted). Based on the extracted features, objects within the open area are identified, resulting in the first object detection result. For example, if an object matching the outline and texture features of a "black leather driver's license" is detected on the center console surface, this object can be identified as the target object, generating the first object detection result: "A black leather driver's license exists on the center console."

[0025] Subsequently, the vehicle's control module also responds to the query request, obtaining the item attribute information of the hidden areas inside the vehicle. Hidden areas refer to enclosed or semi-enclosed spaces that require mechanical operation to access, such as the glove box, armrest box, and door panel storage compartments. The item attribute information is obtained by scanning the corresponding hidden areas with sensors. Different hidden areas are equipped with different sensors. For example, the glove box is equipped with a Hall switch (to detect the status of the door; a magnetic field strength <10mT indicates it is closed and triggers internal detection), a 3×3 distributed pressure array (to detect the weight distribution of items; for example, detecting a weight of 85g can match the weight characteristics of a driver's license), and an infrared camera (to penetrate and obtain the outline texture of items). The door panel storage compartment is equipped with an infrared beam sensor (which can be used to detect whether an item has entered; a signal blockage lasting 100ms indicates that an item has entered), a BLE (Bluetooth Low Energy) beacon array (which can be used to locate the position of items, such as locating an item 0.2m in front of the door panel), and a scene tag sensor (which can be used to identify environmental information, such as in rainy weather, to help infer the possible presence of a folding umbrella). The vehicle system fuses multiple item attribute information (such as weight, location, and environmental information) obtained from the aforementioned sensors. Using appropriate algorithms (such as weight similarity algorithms, contour similarity algorithms, DS evidence theory, and Bayesian networks), it infers whether a target item exists within the hidden area, thus obtaining a second item detection result. For example, when the glove box is closed, the pressure array detects a weight of 85g, and the infrared camera captures an image matching the texture of a driver's license. After fusion calculation using DS evidence theory, the system determines that the probability of a target driver's license existing in the glove box reaches 90.3%, generating a second item detection result: "There is a high probability that a black leather driver's license exists in the glove box."

[0026] Finally, the vehicle's communication module sends the first and second item detection results obtained in the preceding steps to the client for user viewing. This embodiment distinguishes between open and hidden areas and employs targeted detection methods. Utilizing specialized sensors and algorithms, it ensures detection accuracy while fulfilling the user's need for remotely checking items inside the vehicle. This avoids users having to travel back and forth to the vehicle due to uncertainty about item availability, significantly saving time and effectively solving the problems of blind spots, low accuracy, and poor user experience in existing in-vehicle item detection technologies.

[0027] Figure 2 A flowchart of another embodiment of a method for detecting items inside a vehicle according to the present invention is shown, the method being performed by a vehicle. Figure 2 As shown, the method includes the following steps:

[0028] Step S201: Receive a query request sent by the client. The query request is used to query the target item. See details below. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0029] Step S202: In response to the query request, the in-vehicle camera is called to capture images inside the vehicle, and the items in the open areas inside the vehicle are identified through the in-vehicle images to obtain the first item detection result.

[0030] Specifically, step S202 includes:

[0031] Step S2021: Extract image features from the in-vehicle image;

[0032] Step S2022: Based on image features, the object detection model identifies the outlines and textures of objects in open areas inside the vehicle.

[0033] Step S2023: Determine the type of items in the open areas inside the vehicle based on the item outline and item texture;

[0034] Step S2024: When the outline of an item is greater than a preset outline threshold, calculate the first matching relationship between the item type and the target item, and determine the detection result of the first item through the first matching relationship;

[0035] Step S2025: When the outline of an item is smaller than a preset outline threshold, the item type and the interior image of the vehicle are sent to the client so that the client can confirm the second matching relationship between the item type and the target item with the user.

[0036] Step S2026: Receive the confirmation instruction sent by the client and, in response to the confirmation instruction, output the first item detection result. The confirmation instruction is sent by the client when it receives information from the user confirming the second matching relationship.

[0037] Specifically, in this embodiment of the invention, when performing object detection in open areas inside a vehicle, the image captured by the in-vehicle camera is first processed using image preprocessing algorithms (such as grayscale conversion and noise reduction filtering) to extract image features from the in-vehicle image. The image features in this embodiment cover key information such as the color distribution of the object, edge contour details, and surface texture patterns. For example, when a rectangular object is captured on the surface of the center console, the deep black tone features of the object, the regular line features of its surrounding edges, and the leather texture features of its surface are extracted.

[0038] Next, a pre-trained object detection model (such as the YOLOv8 object model, a single-stage object detection algorithm, used as an example only, not as a limitation) is invoked. Based on the extracted image features, objects in open areas inside the vehicle (such as the trunk, seat surfaces, and center console, areas that can be directly observed without mechanical operation) are identified. The model focuses on identifying object outlines (such as the cuboid outline of a suitcase, or the structural outline with a handle and wheels) and object textures (such as the brushed metal texture of a laptop surface, or the smooth glaze texture of a water cup). For example, when identifying items in the trunk, the model can clearly distinguish the combined outline of a 20-inch suitcase with a cuboid shape, handle, and wheels, as well as the checkered fabric texture on the surface of the suitcase.

[0039] Subsequently, based on the object outlines and textures identified in the above steps, the system compares them with the built-in object feature library (which stores outlines and texture templates of common in-vehicle items, such as the rectangular outline of a driver's license, textures with anti-counterfeiting watermarks, the folded outline of a wallet, and leather embossing textures) to determine the type of item in the open area of ​​the vehicle. For example, if the identified object outline is a small rectangle, the texture has an anti-counterfeiting pattern, and the size matches the driver's license template in the feature library, the object type is determined to be a "driver's license".

[0040] Next, the system calls the preset outline threshold (this threshold is set according to the common item size, such as setting 20cm×15cm as the outline threshold for large items and 5cm×3cm as the outline threshold for small items, which can be adjusted according to the actual application scenario), compares the identified item outline size with the preset outline threshold, and calculates the first matching relationship between the item type ("20-inch suitcase") and the target item queried by the user (such as the user querying "22-inch black suitcase") by using the cosine similarity algorithm or the Euclidean distance algorithm. That is, it calculates the similarity between the two in terms of features such as outline, texture, and color. If the similarity reaches 95% or more, the first item detection result is determined to be "a target 20-inch black suitcase exists in the trunk". If the object's outline is smaller than a preset outline threshold (e.g., a small object on the seat surface with an outline size of approximately 4cm × 2cm is detected, which is smaller than the preset threshold of 5cm × 3cm), then step S2025 is executed. The determined object type (e.g., "suspected lipstick") and the corresponding in-vehicle image are sent to the client (e.g., the user's mobile app) through the vehicle's communication module. After receiving the information, the client displays the object type and image to the user in the form of a pop-up window or message notification, guiding the user to confirm the second matching relationship between the object type and the target object (e.g., the user searches for "red lipstick"). For example, after viewing the image and object type prompt, if the user confirms that the object is indeed the "red lipstick" they searched for, then they can click the "Confirm Match" button on the client to avoid the problem of small objects being easily misdetected.

[0041] Finally, the vehicle's communication module receives a confirmation command from the client (this confirmation command is generated and sent by the client according to a preset communication protocol after receiving the user's confirmation of the second matching relationship, and includes key data such as item type and matching result). The system responds to this confirmation command and outputs the final first item detection result (e.g., "Target red lipstick is present on the seat surface"). This embodiment of the invention refines the open area item detection process step-by-step, achieving accurate identification by combining a target detection model. Simultaneously, it sets differentiated processing logic for items of different sizes, ensuring both the efficiency and accuracy of large item detection. Furthermore, the user confirmation mechanism compensates for potential misjudgments of small items due to feature ambiguity, effectively improving the reliability of open area item detection within the vehicle. This further improves the overall in-vehicle item detection system, reduces user confusion caused by item misjudgments, and enhances the user experience of remotely querying items within the vehicle.

[0042] Step S203: In response to the query request, obtain the item attribute information of the hidden area, and infer the detection result of the second item based on the item attribute information. The item attribute information is obtained by scanning the hidden area using a sensor. For details, please refer to [link to details]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0043] Step S204: Send the detection results of the first item and the second item to the client. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0044] Figure 3 A flowchart of another embodiment of a method for detecting items inside a vehicle according to the present invention is shown, the method being performed by a vehicle. Figure 3 As shown, the method includes the following steps:

[0045] Step S301: Receive a query request sent by the client. The query request is used to query the target item. See details below. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0046] Step S302: In response to the query request, the in-vehicle camera is used to capture images of the vehicle interior, and items in open areas inside the vehicle are identified using these images to obtain the first item detection result. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment or Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0047] Step S303: In response to the query request, obtain the item attribute information of the hidden area, and infer the second item detection result through the item attribute information. The item attribute information is obtained by scanning the hidden area sensor.

[0048] Specifically, step S303 includes:

[0049] Step S3031: Obtain attribute information of multiple items in the hidden areas of the vehicle by scanning them with various hidden area sensors;

[0050] Step S3032: Merge the multiple item attribute information obtained from the scan, and calculate the third matching relationship between the items in the hidden area and the target item based on the fusion result;

[0051] Step S3033: Determine the detection result of the second item through the third matching relationship.

[0052] Specifically, in this embodiment of the invention, when performing item detection in a hidden area inside a vehicle to obtain a second item detection result, it first calls upon multiple hidden area sensors pre-configured in the hidden area. Each sensor scans the items within the area using its specific function, acquiring multi-dimensional item attribute information. For example, if the object to be detected is a glove box, a Hall effect switch sensor can detect the open / closed state of the door (determining whether it is in a closed detection state), a distributed pressure sensor array can collect the weight and weight distribution of the items inside the box, and an infrared camera can scan the outline and surface texture of the items, simultaneously acquiring multiple attribute information such as "door closed," "85g item present at a certain location," and "item is rectangular with anti-counterfeiting texture." If the object to be detected is a door panel storage compartment, an infrared beam sensor can detect whether an item has entered, a BLE beacon array can locate the item's position, and a scene tag sensor can identify the current environment (such as whether it is raining), obtaining attribute information such as "item entered," "item located 0.2m in front of the door panel," and "currently raining," achieving comprehensive acquisition of multi-dimensional attributes of items in the hidden area.

[0053] Next, data fusion technology is used to integrate multiple item attribute information obtained from different sensors, eliminating potential biases or contradictions in data from a single sensor. Based on the fused complete information, a third matching relationship (i.e., the similarity between the two or the credibility of the target item's existence) is calculated between items within the hidden area and the target item queried by the user. For example, when the target item is a "driver's license," and the attribute information obtained from the glove box is "85g weight," "rectangular outline," and "with anti-counterfeiting texture," this information is fused and compared with the standard attributes of a driver's license in the system (such as typical weight 80-90g, rectangular shape, and anti-counterfeiting texture) to calculate the matching credibility. If the attribute information obtained from the door storage compartment is "rainy environment," "item entered," and "located in front of the door," and the target item is a "folding umbrella," then the environmental information is fused with the item's location, existence status, and other information, combined with the common scenario of carrying a folding umbrella in the rain, to calculate the matching degree between the item and the folding umbrella, thus obtaining a third matching relationship.

[0054] Finally, based on the calculated third matching relationship and combined with the system's preset judgment rules (such as determining the existence of a target item if the matching confidence reaches a certain threshold, and determining it as suspected or non-existent if it does not reach the threshold), the detection result of the second item is determined. For example, if the third matching relationship shows a confidence level of 90% or higher, and the target item is a driver's license, the detection result of the second item is "a target driver's license exists in the glove box"; if the third matching relationship shows a confidence level of 60%, and the target item is a folding umbrella, the detection result of the second item is "a target folding umbrella is suspected to exist in the door panel storage compartment".

[0055] This invention employs a process of collecting multi-dimensional attributes of objects using multiple sensors, fusing information to calculate matching relationships, and determining detection results based on these matching relationships. This effectively covers the core aspects of hidden area object detection, solves the problem that traditional single detection methods struggle to comprehensively acquire information about hidden area objects, and improves the comprehensiveness and accuracy of hidden area object detection. It provides a reliable hidden area detection solution for full-domain detection of objects inside vehicles, better meeting users' needs for remotely querying hidden area objects within vehicles.

[0056] Step S304: Send the detection results of the first item and the second item to the client. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0057] In some optional implementations, when the hidden region queried by the query request is a semi-closed region, step S3031 includes:

[0058] Step a1: Detect whether an item is placed in the semi-enclosed area using a through-beam infrared sensor to obtain the first attribute information;

[0059] Step a2, and / or, locate the object's position using a beacon sensor to obtain the second attribute information;

[0060] Step a3, and / or, identify environmental information through scene sensors, estimate the types of items that may be carried based on the environmental information, and obtain third attribute information;

[0061] Step a4, and / or, using an in-vehicle camera to photograph the parts of the object exposed outside the semi-enclosed area to obtain the first object feature;

[0062] Step a5, and / or, measuring the weight of the item within the semi-enclosed area using a pressure sensor array to obtain a second item feature.

[0063] Specifically, when the hidden area pointed to by the query request is a semi-enclosed area (such as a door panel storage compartment in a vehicle, a semi-open storage bag on the side of the seat, or other areas that require simple mechanical operation and have some exposed space), the embodiments of the present invention obtain multi-dimensional attribute information of items in the semi-enclosed area through the combined detection of multiple types of sensors.

[0064] In this embodiment, through-beam infrared sensors are deployed on both sides of the entrance to the semi-enclosed area (such as the upper and lower edges of the door panel storage compartment opening). These sensors operate by continuously emitting infrared beams from the transmitter and receiving the beams in real time. When an item is placed into or removed from the semi-enclosed area, the item blocks the infrared beam, causing a change in the intensity of the light signal received by the receiver. The system determines whether an item has been placed in the area by monitoring the change in light signal intensity. For example, if the receiver does not detect an infrared beam for 100ms (i.e., the beam is continuously blocked), it determines that an item has been placed and generates the first attribute information "an item exists in the semi-enclosed area"; if the receiver continuously detects an infrared beam, it determines that no item exists in the area, and the first attribute information is "no item in the semi-enclosed area".

[0065] Furthermore, beacon sensors (such as BLE beacon sensors) can be deployed on objects within semi-enclosed areas via adhesive or attachment. These sensors interact with vehicles by emitting wireless signals at specific frequencies, calculating the object's precise location within the area based on signal propagation time differences or signal strength attenuation patterns. This generates secondary attribute information, such as "the object is located 0.2m in front of the semi-enclosed area." If no valid signal is detected, no secondary attribute information is generated, or the location is marked as "object location pending confirmation."

[0066] Furthermore, in this embodiment of the invention, a scene sensor can be deployed near the vehicle's center console or a semi-enclosed area. This sensor can collect environmental parameters (such as outdoor precipitation, in-vehicle temperature, seasonal correlation data, etc.) and match these environmental parameters with a preset scene-item association model to estimate the types of items the user might be carrying in the current scene. For example, if the scene sensor detects outdoor precipitation > 5 mm / h (determined as a rainy day scene), it combines the association rule in the model that "users often carry folding umbrellas and waterproof bags in rainy days" to generate third attribute information such as "possible items like folding umbrellas and waterproof bags"; if it detects an in-vehicle temperature < 5℃ (determined as a low-temperature scene), it generates third attribute information such as "possible items like hand warmers and scarves".

[0067] In addition, the camera angle of the in-vehicle camera (such as a 1080P infrared camera mounted on the roof) can be adjusted to cover exposed parts of a semi-enclosed area (such as the edges or handles of items exposed at the opening of a door storage compartment). Images of these exposed parts can be captured by the camera, and local features of the items can be extracted from the images (such as the texture of an exposed umbrella handle or the color of an exposed cup lid). These local features are then used as the first item feature. For example, if a black cylindrical component with anti-slip texture is captured at the opening of a semi-enclosed area, the first item feature would be "a black cylindrical object with anti-slip texture."

[0068] In addition, a pressure sensor array (such as a 3×3 distributed pressure sensor) can be installed at the bottom of the semi-enclosed area. This array detects the pressure value generated when an item is placed by each sensor unit, and calculates the overall weight and weight distribution of the item by combining the positional distribution of the sensor units. For example, if a sensor unit in the middle of the pressure sensor array detects 80g of pressure, while the surrounding units do not detect significant pressure, the weight of the item is determined to be approximately 80g, generating a second item characteristic of "item weight approximately 80g, concentrated in the middle of the area". If multiple sensor units detect pressure, the shape of the item can be inferred by combining the sum and distribution pattern of the pressure values ​​of each unit (e.g., a long strip-shaped item will cause multiple sensor units to generate pressure along its length).

[0069] By scanning multiple hidden area sensors to obtain the attribute information of various items in the hidden areas of the vehicle (such as quickly confirming the existence, location and possible type of items, or selecting to obtain the existence, local features and weight of items based on the structural characteristics of the semi-enclosed area), the attribute information of items in the semi-enclosed area can be comprehensively collected. This provides rich and multi-dimensional data support for subsequent item attribute information fusion and third-party matching relationship calculation, effectively solving the problem of incomplete detection information caused by the "partial enclosure and partial exposure" of the semi-enclosed area, improving the accuracy of item detection in the semi-enclosed area, and further improving the detection system for hidden areas in the vehicle.

[0070] In some optional implementations, based on steps a1 to a5 above, step S3032 includes:

[0071] Step b1: Based on the first attribute information, the second attribute information, and the third attribute information, respectively, infer the first probability, the second probability, and the third probability that the target item is located in the semi-enclosed area;

[0072] Step b2: Based on the Bayesian estimation method, the first probability, the second probability and / or the third probability are fused to obtain the confidence level of the target item's location in the semi-enclosed area;

[0073] Step b3: When no first or second item features are collected, the location confidence score is used as the third matching relationship.

[0074] Step b4: When the first item feature and / or the second item feature are collected, estimate the type confidence of belonging to the target item based on the first item feature and / or the second item feature, and use the type confidence and the location confidence as the third matching relationship.

[0075] Specifically, this embodiment of the invention, based on the first attribute information, second attribute information, and third attribute information obtained in the semi-enclosed area detection, respectively infers the first probability, second probability, and third probability that the target item is located within the semi-enclosed area. The first attribute information is the result of the through-beam infrared sensor detecting "an item exists within the semi-enclosed area," combined with historical data on "the probability that the target item actually exists when the through-beam infrared sensor detects the item" (e.g., statistically, the historical probability of the target item existing after the sensor detects the item is 90%), and the first probability is set as a statistical value, for example, 90%. The second attribute information is the location of the item located in the semi-enclosed area by the beacon sensor. Referring to the overlap between this location and the high-frequency placement location of the target item (e.g., when the target item is a folding umbrella, the high-frequency placement overlap in front of the door storage compartment is 85%), a second probability can be defined, for example, set to 85%. The third attribute information is the probability that the user might be carrying the type of item identified by the scene sensor in the current scene. For example, the "possible presence of a folding umbrella" estimated after "rainy weather environment" is based on the historical probability of a user carrying a folding umbrella and placing it in the semi-enclosed area in a rainy weather scenario (e.g., the probability of placing a folding umbrella in the area on a rainy day is 75%), and the prior third probability is defined as 75%.

[0076] Then, a Bayesian estimation method was used to calculate the posterior probability, which involves fusing the first, second, and third probabilities to obtain the confidence level of the target item's location within the semi-enclosed region. The Bayesian estimation method uses the probabilities corresponding to each attribute information as prior probabilities, calculates the joint probability using the probability multiplication formula, and then combines normalization to eliminate probability overflow issues. Specifically, the first probability P1=90%, the second probability P2=85%, and the third probability P3=75% were first converted to decimal form (i.e., 0.9, 0.85, 0.75), and the joint probability P=P1×P2×P3=0.9×0.85×0.75=0.57375 was calculated. Since the attribute information is independent, no additional conditional probability correction is needed; this joint probability was directly used as the initial location confidence level. After verification with a preset probability range (0-1), the final location confidence level was determined to be 57.375%.

[0077] Subsequently, depending on whether the first and second item features were collected, the following steps are executed. If the first item feature was not collected (i.e., the in-vehicle camera did not capture the exposed part of the item outside the semi-enclosed area) and the second item feature was not collected (i.e., the pressure sensor array did not measure the weight of the item within the semi-enclosed area), the calculated location confidence score (57.375%) is directly used as the third matching relationship, indicating the possibility that the target item exists in the semi-enclosed area. If the first item feature and / or the second item feature are collected, for example, the first item feature of "black cylindrical shape with anti-slip texture" exposed at the opening of a semi-enclosed area captured by the in-vehicle camera (matching the features of a folding umbrella handle), and the second item feature of "item weight approximately 350g" measured by the pressure sensor array (matching the typical weight of a folding umbrella), then the similarity between the first item feature and the target item (folding umbrella) is further calculated (e.g., the matching degree of "black cylindrical shape with anti-slip texture" with the folding umbrella handle feature is 92%), combined with the fit between the second item feature and the target item weight (the overlap between 350g and the standard weight of a folding umbrella, 300-400g, is approximately 92%). The type confidence is calculated using a weighted average algorithm (feature similarity weight 0.6, weight fit weight 0.4), which is 92% × 0.6 + 100% × 0.4 = 95.2%. This type confidence (95.2%) and location confidence (57.375%) are then used together as the third matching relationship to form a two-dimensional matching result with a type confidence of 95.2% and a location confidence of 57.375%. This indicates the probability that the target item exists in the semi-enclosed area and that the target item is of a certain type. For example, there is a 57.375% probability that an item is in the semi-enclosed area and a 95.2% probability that the item is an umbrella.

[0078] This invention, through step-by-step probability calculation, confidence fusion, and supplementary correction based on item features, fully utilizes multi-source detection data from semi-enclosed areas and achieves scientific probability fusion through Bayesian estimation. Furthermore, it flexibly adjusts the composition of the third matching relationship based on the presence or absence of item features, effectively solving the problem of insufficient matching accuracy caused by partial exposure and information fragmentation in semi-enclosed areas. This significantly improves the accuracy and reliability of target item detection within semi-enclosed areas, providing precise algorithmic support for differentiated detection of hidden areas inside vehicles and further meeting users' remote query needs for items in semi-enclosed areas within vehicles.

[0079] In some optional implementations, when the hidden region queried by the query request is a first fully enclosed region, the first fully enclosed region is used to represent a frequently opened fully enclosed region. Step S3031 includes:

[0080] Step c1: Collect the weight distribution in the first fully enclosed area using a pressure sensor array to obtain the third item characteristics;

[0081] Step c2: When the first fully enclosed area is opened, the outline and texture of the objects within the first fully enclosed area are captured by the in-vehicle camera to obtain the fourth object feature;

[0082] Step c3, and / or, using a beacon sensor to locate the position of an object within the first fully enclosed area to obtain fourth attribute information.

[0083] Specifically, when the hidden area pointed to by the query request is the first fully enclosed area (i.e., a fully enclosed storage space in the vehicle that is frequently used by the user and needs to be opened / closed by mechanical operation, such as the armrest box in the vehicle, the storage box under the front seat, etc., such areas need to balance detection efficiency and data integrity due to the high frequency of opening), the embodiments of the present invention deploy sensor combinations by adapting to the "frequent opening" characteristics of the first fully enclosed area.

[0084] In this first fully enclosed area, a pressure sensor array (such as a 3×3 or 4×4 distributed pressure sensor unit, with a single unit detection accuracy of up to 1g) is laid on the bottom of the inner wall (e.g., the bottom of the armrest box or the bottom of the storage box under the seat). This pressure sensor array is electrically connected to the vehicle control module and can collect pressure distribution data generated when items are placed in the area in real time. When an item is placed in the first fully enclosed area, different units of the pressure sensor array will generate corresponding pressure values ​​according to the contact position and weight of the item. The control module summarizes and analyzes these pressure values ​​to generate information including the total weight of the item and the area of ​​concentrated weight, which serves as a third item characteristic. For example, when a user places a wallet into the armrest box, the three adjacent sensing units in the middle of the pressure sensor array detect pressure values ​​of 25g, 30g, and 28g respectively, while the other units provide no pressure feedback. The control module calculates the total weight to be 83g, and the weight is concentrated in the middle area of ​​the armrest box. Thus, the third item characteristic is "total weight 83g, weight concentrated in the middle area". If multiple items are placed in the area (such as a wallet and keys placed at the same time), the pressure sensor array will detect multiple sets of discrete pressure values. The control module can distinguish the weight and placement of different items based on the distribution range of the pressure values, forming a third item characteristic corresponding to multiple items. For example, [0, 0, 0] indicates no items in the front, [5, 85, 0] indicates an 85g item in the middle, with the edge of the item slightly touching the left-center sensor, resulting in a 5g detection result, and [0, 15, 0] indicates a 15g item in the middle of the back.

[0085] Outside the first fully enclosed area, an in-cabin camera (e.g., a 180° wide-angle camera) is installed inside the cabin. This camera's field of view covers the outside of the first fully enclosed area and only starts capturing images when the first fully enclosed area is open. The opening of the first fully enclosed area can be detected by Hall effect sensors installed at the drawer opening and cover. When the first fully enclosed area is opened, the camera immediately starts and captures images of the items within the area. The control module preprocesses the captured images (e.g., noise reduction and grayscale conversion) and extracts the outline shape of the items in the images. For example, the camera in the seat drawer is installed at the bottom of the seat (external). When the drawer is open, the wide-angle camera (180° field of view) at the bottom of the seat captures the interior space of the drawer by shooting downwards at an angle, capturing the outline texture of the items (e.g., the rectangular outline and leather texture of a wallet). This captured information is then used as the fourth item feature. For example, if the camera captures a rectangular item in the armrest box with an outline size of approximately 12cm × 8cm and an irregular leather texture on the surface, the fourth item feature is obtained as "rectangular outline (12cm × 8cm), surface leather texture".

[0086] Furthermore, beacon sensors, such as those using UWB (Ultra-Wide Band) micro-tags, can be attached to items in the first fully enclosed area. These beacon sensors have a signal coverage range adapted to the size of the seat drawer and can be pre-attached to frequently used items like wallets and keys. The beacon sensor communicates with the vehicle control module. The vehicle can receive the wireless signals emitted by the UWB micro-tags and calculate the item's exact location within the drawer based on the signal propagation time difference and intensity attenuation, providing the fourth attribute information. For example, if a UWB micro-tag is attached to a wallet, and the vehicle detects a signal from the wallet corresponding to a distance sensor position 0.1m to the left of the drawer, the fourth attribute information is that the item is located 0.1m to the left of the drawer.

[0087] This invention, through real-time acquisition of weight distribution, capture of panoramic contour texture using a wide-angle camera when the drawer is opened, and assistance from a UWB beacon sensor to locate the position of frequently accessed items, comprehensively acquires the weight, shape, texture, and position information of items within the first fully enclosed area. This provides a multi-dimensional and highly reliable data foundation for subsequent item attribute information fusion and third-party matching relationship calculation. It not only adapts to the "frequent opening" characteristic of the first fully enclosed area but also improves the completeness of information acquisition through multi-sensor data complementarity. It effectively solves the problems of fragmented and untimely item detection data in frequently opened fully enclosed areas, providing reliable support for accurate item detection in the first fully enclosed area.

[0088] In some optional implementations, based on steps c1 to c3 above, step S3032 includes:

[0089] Step d1: Create an item feature vector based on the fourth attribute information, the third item feature, and the fourth item feature;

[0090] Step d2: Obtain the standard feature vector of the target item;

[0091] Step d3: The third matching relationship is obtained by calculating the distance between the item feature vector and the standard feature vector.

[0092] Specifically, within the first fully enclosed area pointed to by the query request, a third matching relationship needs to be calculated based on the fourth attribute information, the third item features, and the fourth item features. This embodiment of the invention combines the K-nearest neighbor algorithm with a dynamic feature library mechanism to solve the problem of difficulty in distinguishing similar items (such as wallets and purses) when multiple items are stacked in a small space.

[0093] First, based on the fourth attribute information, third item features, and fourth item features obtained from the detection of the first fully enclosed area, an item feature vector is created. For example, for the item to be detected in the drawer, the created item feature vector is X=[50,1.0,1.0,0.10] (corresponding to weight, contour matching degree, texture similarity, and position, respectively). If there are multiple items stacked in the first fully enclosed area (such as wallets and lipsticks stacked), the system will distinguish the feature information of different items by the weight distribution difference of the pressure sensor array (wallets weighing 50g are concentrated on the left side, and lipsticks weighing 15g are concentrated on the right side) and the contour edge recognition of the camera, and create feature vectors for the corresponding items to avoid feature confusion.

[0094] The system then retrieves the standard feature vector of the target item from the vehicle's local dynamic feature library. This dynamic feature library stores historical feature templates of items the user carries daily and supports iterative updates based on user feedback (i.e., after the user confirms the detection result, the system automatically adds new item features to the feature library). For the target item "black wallet," the feature library already stores historically confirmed standard feature vector templates, such as X0=[55,1.0,1.0,0.12] (corresponding to weight 55g, contour matching degree 1.0 (rectangle), texture similarity 1.0 (leather), and location 0.12m (0.12m from the left side of the drawer), which is feature data of "black wallet" previously confirmed by the user and has high reference reliability. If the target item is queried for the first time and there is no corresponding standard template in the feature library, the system will call the cloud-based general item feature library to obtain the typical standard feature vector of this type of item. For example, the standard feature vector of a general black wallet is [52±3,1.0,1.0,0.11±0.02], which is used as a temporary standard feature vector for subsequent calculations.

[0095] Finally, the core calculation logic of the K-nearest neighbors algorithm is used to calculate the Euclidean distance between the item's feature vector and the standard feature vector, thereby obtaining the third matching relationship (i.e., the matching degree between the item and the target item). A specific calculation process is illustrated below:

[0096] Taking the item feature vector X=[50,1.0,1.0,0.10] and the standard feature vector X0=[55,1.0,1.0,0.12] as an example, first calculate the difference between the data in each dimension and square it:

[0097] Weight dimension: (normalized (50-55)) 2 =(-0.1) 2

[0098] Contour matching degree dimension: (1.0-1.0) 2 =0

[0099] Texture similarity dimension: (1.0-1.0) 2 =0

[0100] Location dimension: (normalized (0.10-0.12)) 2 =(0.05) 2

[0101] Then sum the squared values ​​of each dimension and take the square root to obtain the Euclidean distance. In engineering applications, it is simplified to 0.15 (the decimal result is reasonably simplified for ease of calculation and judgment in engineering applications).

[0102] Matching degree conversion: The normalized Euclidean distance is converted into a matching degree using the formula "matching degree = 1 - normalized Euclidean distance". Substituting the data, we get:

[0103] Matching degree = 1 - 0.15 = 0.85 (i.e. 85%). This matching degree is the third matching relationship. The system determines that the credibility of the current item as "black wallet" is 85%.

[0104] If a user views the detection results through a client (such as a mobile app) and clicks "Confirm Black Wallet," the system will respond to this confirmation command by adding the detected item's feature vector X=[50,1.0,1.0,0.10] to the dynamic feature library as a supplementary standard template for black wallets, thus updating the dynamic feature library. The next time an item with the feature vector [50,1.0,1.0,0.10] is detected, its Euclidean distance to the newly added standard template is calculated to be 0, indicating a 100% match. It can then be directly identified as a "black wallet" without requiring further user confirmation, thus achieving an iterative improvement in detection accuracy.

[0105] This invention combines the distance calculation logic of the K-nearest neighbor algorithm with the iterative mechanism of the dynamic feature library. It can accurately quantify the feature differences between the item and the target item, solving the problem of difficulty in distinguishing similar items when multiple items are stacked in a small space. It can also continuously optimize the feature library through user feedback, improving the efficiency and accuracy of subsequent detection. This provides reliable algorithmic support for the accurate detection of items in the first fully enclosed area and further improves the intelligent detection system for items in the vehicle.

[0106] In some optional implementations, when the hidden region queried by the query request is a second fully enclosed region, the second fully enclosed region is used to represent a fully enclosed region that is not frequently opened. Step S3031 includes:

[0107] Step e1: Collect the weight distribution in the second fully enclosed area using a pressure sensor array to obtain the fifth item feature;

[0108] Step e2: The outline and texture of the objects within the second fully enclosed area are captured by an infrared camera within the second fully enclosed area to obtain the sixth object feature.

[0109] Specifically, when the hidden area pointed to by the query request is the second fully enclosed area (i.e., a fully enclosed storage space in the vehicle that is not frequently opened by the user and is in a closed state for a long time, such as the hidden storage box on the side of the trunk, the sealed storage compartment on the back of the rear seat, the glove box, etc. Because such areas are not frequently opened, they need to be accurately detected in a closed state to avoid the cumbersome operation caused by frequent opening), the sensor combination adapted to the "not frequently opened" characteristic of the second fully enclosed area is combined with the trigger control logic of the Hall switch to collect the attribute information of the items in the area step by step.

[0110] First, a Hall effect switch (with a magnetic field detection threshold of 10mT and a response time ≤10ms) is deployed at the junction of the door and the enclosure in the second fully enclosed area. This switch acts as a trigger for the pressure sensor array and infrared camera detection, determining whether the area is in a suitable closed state for detection. When the Hall effect switch detects a magnetic field strength <10mT at the door (e.g., an actual detection value of 8mT), the system initiates a 50ms anti-shake verification process. If the magnetic field strength is consistently below 10mT for three consecutive measurements, the second fully enclosed area is confirmed to be sealed, and the detection process of the pressure sensor array and infrared camera is triggered. If the Hall effect switch detects a magnetic field strength >10mT, the door is determined to be open, and only the current state information is recorded; subsequent detection is not initiated.

[0111] A 3×3 distributed pressure sensor array is installed at the bottom of the second fully enclosed area. This array is electrically connected to the vehicle control module and collects pressure data generated by the placement of objects within the area in real time after the Hall switch triggers the detection process. The vehicle control module receives the pressure matrix data transmitted by the pressure sensor array, analyzes the data, determines the object placement area based on the location of non-zero pressure values, and calculates the object weight by summing the non-zero pressure values, forming a fifth object feature that includes the object weight and its distribution location. For example, the pressure matrix collected by the pressure sensor array is [[0,0,0],[5,85,0],[0,15,0]], where the first row represents the front of the area, the second row represents the middle, and the third row represents the rear. The three values ​​in each row correspond to the pressure values ​​of the left, middle, and right sensing units, respectively. The control module analyzes and determines that there is an 85g object in the middle of the middle of the area (with the edge slightly contacting the left sensing unit in the middle, generating 5g pressure), and a 15g object in the middle of the rear of the area. Therefore, the fifth object feature is generated as "Middle of the middle: 85g object, Middle of the rear: 15g object". If there is only a single item in the area, the sensing unit corresponding to the item's placement position in the pressure matrix will generate a non-zero pressure value. The control module will directly calculate the sum of these non-zero values ​​as the item's weight, forming the fifth item feature of the single item.

[0112] Furthermore, in this embodiment of the invention, an infrared camera (using an 850nm wavelength infrared light source, capable of penetrating slightly obstructed objects, with a resolution of 1080P, and a shooting angle covering the entire internal space of the area) is installed on the inner top or side wall of the second fully enclosed area. This infrared camera is activated synchronously after the Hall switch triggers the detection process, capturing infrared images of the items within the area. Since the second fully enclosed area is in a closed state for a long time and the light is dim, the infrared camera illuminates the interior of the area through its built-in infrared supplementary lighting module, clearly capturing the outline shape and surface texture of the items. Even if there is slight dust or a thin cloth covering the surface of the items, the infrared light can penetrate and present the true outline of the items. After the vehicle control module preprocesses the infrared image (such as noise reduction and edge enhancement), it extracts the outline (such as the rectangular outline of a driver's license and the cylindrical outline of a lipstick) and surface texture (such as the anti-counterfeiting watermark texture of a driver's license and the smooth glaze texture of a lipstick shell) of each item in the image, and integrates this information into the sixth item feature.

[0113] Through the aforementioned technical means, the weight distribution, outline, and texture information of items within the second fully enclosed area can be comprehensively acquired even when the area remains closed. This not only adapts to the "infrequently opened" nature of the second fully enclosed area (detection can be completed without the user opening the area, reducing operational costs), but also avoids the limitations of single-sensor detection through the complementary detection of pressure sensor array and infrared camera. This provides accurate and complete data support for subsequent fusion of item attribute information and calculation of third-party matching relationships, effectively solving the problem of blind spots in item detection caused by the "long-term closure and inconvenience of detection" in infrequently opened fully enclosed areas, and further improving the full-domain detection system for hidden areas inside the vehicle.

[0114] In some optional implementations, based on steps e1 to e2 above, step S3032 includes:

[0115] Step f1: Based on the characteristics of the fifth item and the weight similarity of the target item, calculate the first evidence that the target item exists within the second fully enclosed area;

[0116] Step f2: Based on the characteristics of the sixth item and the shape similarity of the target item, calculate the second evidence that the target item exists within the second fully enclosed area;

[0117] Step f3: The first and second pieces of evidence are fused using an evidence theory algorithm to obtain the third matching relationship.

[0118] Specifically, in the second fully enclosed area pointed to by the query request, and after the features of the fifth and sixth items have been obtained, this embodiment of the invention uses the DS evidence theory algorithm to calculate the third matching relationship of the target item step by step, effectively solving the problem of contradictory data from a single sensor when multiple items are stacked in an enclosed space.

[0119] This invention, based on the obtained fifth item feature (i.e., item weight distribution data collected by a 3×3 distributed pressure sensor array), compares it with the standard weight feature of the target item to calculate the weight similarity, thereby obtaining the first evidence that the target item exists within the second fully enclosed area. Taking the target item as a "driver's license" as an example, the system's built-in standard weight feature for a driver's license is 80~90g. The fifth item feature shows that there is an 85g item in the middle of the second fully enclosed area. The credibility of the first evidence is set by weight similarity. The credibility of the target item "exists" is m_A(exists) = 0.8 (because the weight perfectly matches the standard range, but a single weight data may be confused with other items, so it is not set to 1.0), and the credibility of the target item "does not exist" is m_A(does not exist) = 0.2 (reserving other possibilities). That is, the first evidence is m_A(exists) = 0.8 and m_A(does not exist) = 0.2.

[0120] Next, based on the sixth item feature (i.e., the item outline and texture data extracted by the infrared camera), it is compared with the standard shape features (including outline size and surface texture) of the target item to calculate the shape similarity, thus obtaining the second evidence of the target item's existence. Taking the target item "driver's license" as an example, its standard shape features are "rectangular outline (8.5cm × 5.4cm) with anti-counterfeiting watermark texture on the surface." The sixth item feature shows that the outline of the item in the middle of the second fully enclosed area is rectangular (size 8.4cm × 5.5cm, with an error of ≤2% from the standard size), and an irregular anti-counterfeiting watermark texture is extracted from the surface (texture matching degree is calculated to be 92% by the image feature comparison algorithm). Combining the outline matching degree (98%) and the texture matching degree (92%), the weighted average (outline weight 0.4, texture weight 0.6) is taken to obtain the shape similarity = 98% × 0.4 + 92% × 0.6 = 94.4%. Based on the penetration imaging accuracy of the infrared camera, the credibility of the second piece of evidence is set. The credibility of the target item "exists" is m_B(exists) = 0.7 (because the similarity in appearance is high, but there may be similar-looking items (such as bank cards) in the enclosed space, so a certain degree of uncertainty is retained). The credibility of the target item "does not exist" is m_B(does not exist) = 0.3. That is, the second piece of evidence is m_B(exists) = 0.7 and m_B(does not exist) = 0.3.

[0121] Finally, the DS evidence theory algorithm is used to fuse the first and second pieces of evidence to calculate the third matching relationship, which is the final credibility of the existence of the target item.

[0122] The conflict coefficient K measures the degree of contradiction between two pieces of evidence, and is calculated using the formula: "K = m_A(exists) × m_B(does not exist) + m_A(does not exist) × m_B(exists)". Substituting the credibility data of the first and second pieces of evidence, K = 0.8 × 0.3 + 0.2 × 0.7 = 0.24 + 0.14 = 0.38, indicating that there is some conflict between the two pieces of evidence (e.g., the weights match perfectly, but there is slight uncertainty in the matching of appearances), and the conflict needs to be eliminated through evidence synthesis.

[0123] Using the DS evidence synthesis formula, the final credibility is calculated as "m(existence) = [m_A(existence) × m_B(existence)] / (1-K)". Substituting the data, we get: m(existence) = (0.8 × 0.7) / (1-0.38) = 0.56 / 0.62 ≈ 0.903 (i.e. 90.3%). This value is the third matching relationship, indicating that the credibility of the target item "driver's license" existing in the second fully enclosed area is 90.3%.

[0124] In this embodiment of the invention, if a user views the detection results through a client (such as a mobile APP) and clicks "Confirm the existence of a driver's license", the system will respond to the confirmation command and solidify the feature combination of "85g weight, rectangular outline, and anti-counterfeiting texture" detected in this test into the local standard feature library as a new standard template for "driver's license". When items with the same feature combination are detected again in the future, there is no need to recalculate the evidence fusion. The template can be directly matched, and the credibility of the third matching relationship can be improved to more than 95%, which greatly improves the detection efficiency and accuracy.

[0125] By using the technical means provided in this invention, combined with the ability of DS evidence theory to fuse conflicting data from two types of sensors, namely weight and shape, it can effectively solve the problem of single data contradiction caused by multiple items stacked in the second fully enclosed area. It can also continuously improve the detection accuracy by iteratively optimizing the feature library through user feedback, providing reliable algorithm support for the detection of items in fully enclosed areas that are not frequently opened, and further improving the intelligent detection system for items in the vehicle.

[0126] In some optional embodiments, the in-vehicle item detection method provided by the present invention further includes the following steps:

[0127] Step g1: Receive feature update information sent by the client. The feature update information is the information fed back by the client user after confirming the target item based on the second item detection result.

[0128] Step g2, in response to feature update information, updates the object features collected for the hidden area to the standard feature library.

[0129] Specifically, in order to achieve dynamic optimization of the standard feature library and improve the accuracy and efficiency of subsequent hidden area item detection, the embodiments of the present invention also dynamically update the standard feature library.

[0130] The vehicle's communication module maintains real-time data interaction with the client (such as the user's mobile in-vehicle control app). When the user views the detection results of the second item in the hidden area (including the semi-enclosed area, the first fully enclosed area, and the second fully enclosed area) through the client, if the detection result is confirmed to match the actual item (e.g., the client displays "The confidence level of the presence of a driver's license in the second fully enclosed area is 90.3%", and the user clicks the "Confirm Existence" button after verifying the actual item), the client will automatically generate feature update information. This feature update information contains complete item feature data collected during this hidden area detection. For semi-enclosed area detection, the feature data includes the presence status of objects associated with the through-beam infrared sensor, the location of the beacon sensor, the type of environmentally compatible object associated with the scene sensor, the features of exposed parts captured by the camera, and the weight collected by the pressure sensor array. For first fully enclosed area detection (such as a seat drawer), the feature data includes the weight distribution collected by the 10×10 pressure sensor array, the contour texture extracted by the 180° wide-angle camera, and the coordinates of the UWB beacon sensor. For second fully enclosed area detection, the feature data includes the weight collected by the 3×3 pressure sensor array and the contour texture captured by the infrared camera. The client transmits the above feature update information to the vehicle in a encrypted manner according to a preset data format (such as JSON format). After receiving the feature update information, the vehicle communication module transmits it to the control module for data verification (verifying data integrity and format correctness) to ensure that the feature update information can be used for subsequent feature library updates.

[0131] The vehicle control module, responding to the verified feature update information, calls the write interface of the standard feature library to update the feature data of the items collected for the hidden area into the standard feature library. The standard feature library is stored in the vehicle's local storage (such as an onboard solid-state drive), categorized by hidden area type and item type, and supports the addition and updating of feature templates. The control module first determines whether the item type corresponding to the feature update information already has a historical template in the standard feature library. If it is a newly confirmed item type, a new feature template is created in the corresponding category directory, and the data from the feature update information is written into the template. If it is an item type with an existing historical template, the data in the current feature update information is compared with the historical template. If new feature dimensions or feature data optimizations exist, the corresponding fields of the historical template are updated to form a more comprehensive feature template. This embodiment of the invention ensures that the standard feature library always remains consistent with the actual item features collected by the user, continuously improving the accuracy and efficiency of hidden area item detection.

[0132] The technical solution provided by this invention has the following advantages:

[0133] (1) Achieve full coverage detection of items inside the vehicle and eliminate blind spots. Achieve full coverage detection from open areas to hidden areas, completely eliminating the blind spots of traditional technologies, allowing users to remotely confirm whether items have been left in all areas of the vehicle.

[0134] (2) Improve the accuracy of item detection in hidden areas and adapt to the characteristics of different areas.

[0135] In semi-enclosed areas, a combination of infrared beams, beacon positioning, and scene sensors, along with a Bayesian network to dynamically adjust the detection probability, solves the problem of missing feature databases for temporary items (such as folding umbrellas in the rain).

[0136] The first fully enclosed area (frequently opened seat drawers, etc.) employs the K-nearest neighbor algorithm combined with a dynamic feature library, adapted to its characteristics of "frequent opening and easy stacking of similar items." Due to the high frequency of user opening in this area, similar items are easily stacked, making it difficult for traditional algorithms to distinguish them. The K-nearest neighbor algorithm, however, calculates the Euclidean distance between the feature vector of the item to be detected (weight, contour, texture, position) and the standard feature vector, accurately quantifying feature differences. Furthermore, it iteratively updates the feature library based on user feedback, solving the problem of distinguishing similar items while improving detection efficiency through dynamic upgrades.

[0137] The second fully enclosed area (such as the side storage compartment of the trunk, which is not frequently opened): DS evidence theory is used to fuse data from a pressure sensor array and an infrared camera, adapting to its characteristics of "long-term closure and the potential for data inconsistencies due to the accumulation of multiple items." Because this area is not frequently opened, various items tend to accumulate inside, potentially leading to conflicting data from a single sensor. DS evidence theory, by defining evidence sources, calculating conflict coefficients, and synthesizing final credibility, can effectively fuse conflicting data, avoiding misjudgments based on single data points and ensuring the accuracy of detection results under closed conditions.

[0138] (3) Optimize user experience and reduce time costs.

[0139] Figure 4 A schematic diagram of an embodiment of a vehicle interior item detection device of the present invention is shown. The device includes:

[0140] The request receiving module 401 is used to receive query requests sent by the client, which are used to query the target item.

[0141] The open area detection module 402 is used to respond to a query request by calling the in-vehicle camera to capture images inside the vehicle, and to identify items in the open areas inside the vehicle through the images to obtain the first item detection result.

[0142] The hidden area detection module 403 is used to obtain the item attribute information of the hidden area in response to the query request, and to infer the second item detection result through the item attribute information. The item attribute information is obtained by scanning the hidden area sensor.

[0143] The detection result feedback module 404 is used to send the detection results of the first item and the detection results of the second item to the client.

[0144] In one alternative embodiment, the open region detection module 402 includes:

[0145] The feature extraction unit is used to extract image features from in-vehicle images;

[0146] The first appearance recognition unit is used to identify the outline and texture of objects in open areas inside the vehicle based on image features using a target detection model;

[0147] A type recognition unit is used to determine the type of items in open areas inside the vehicle based on the item's outline and texture.

[0148] The first result unit is used to calculate the first matching relationship between the item type and the target item when the item outline is greater than the preset outline threshold, and to determine the first item detection result through the first matching relationship.

[0149] The confirmation unit is used to send the item type and the interior image of the vehicle to the client when the item outline is smaller than a preset outline threshold, so that the client can confirm the second matching relationship between the item type and the target item with the user.

[0150] The second result unit is used to receive the confirmation instruction sent by the client and, in response to the confirmation instruction, output the first item detection result. The confirmation instruction is sent by the client when it receives information from the user confirming the second matching relationship.

[0151] In one alternative embodiment, the hidden region detection module 403 includes:

[0152] The hidden attribute detection unit is used to acquire attribute information of multiple items in the hidden areas of the vehicle scanned by various hidden area sensors.

[0153] The fusion unit is used to fuse multiple item attribute information obtained from scanning, and calculate the third matching relationship between items in the hidden area and the target item based on the fusion result;

[0154] The calculation unit is used to determine the detection result of the second item through a third matching relationship.

[0155] In one alternative approach, when the hidden area queried by the query request is a semi-enclosed area, the hidden attribute detection unit includes: detecting whether an item is placed in the semi-enclosed area using a through-beam infrared sensor to obtain first attribute information; and / or locating the item's position using a beacon sensor to obtain second attribute information; and / or identifying environmental information using a scene sensor and estimating the type of item that may be carried using the environmental information to obtain third attribute information; and / or capturing images of the part of the item exposed outside the semi-enclosed area using an in-vehicle camera to obtain first item characteristics; and / or measuring the weight of the item within the semi-enclosed area using a pressure sensor array to obtain second item characteristics.

[0156] In one optional approach, the fusion unit includes: inferring a first probability, a second probability, and a third probability that the target item is located within a semi-enclosed region based on first attribute information, second attribute information, and third attribute information, respectively; fusing the first probability, the second probability, and / or the third probability using a Bayesian estimation method to obtain a location confidence score that the target item is located within the semi-enclosed region; when no first item feature or second item feature is collected, using the location confidence score as a third matching relationship; when the first item feature and / or second item feature is collected, estimating a type confidence score that belongs to the target item based on the first item feature and / or the second item feature, and using the type confidence score and the location confidence score as a third matching relationship.

[0157] In one alternative approach, when the hidden area queried by the query request is a first fully enclosed area, the hidden attribute detection unit includes: acquiring the weight distribution within the first fully enclosed area through a pressure sensor array to obtain a third item feature; when the first fully enclosed area is opened, capturing the outline and texture of the item within the first fully enclosed area through an in-vehicle camera to obtain a fourth item feature; and / or, locating the position of the item within the first fully enclosed area through a beacon sensor to obtain fourth attribute information.

[0158] In one alternative approach, the fusion unit includes: creating an item feature vector based on fourth attribute information, third item features, and fourth item features; obtaining a standard feature vector of the target item; and obtaining a third matching relationship by calculating the distance between the item feature vector and the standard feature vector.

[0159] In one alternative approach, when the hidden area queried by the query request is a second fully enclosed area, the second fully enclosed area is used to represent a fully enclosed area that is not frequently opened. The hidden attribute detection unit includes: collecting the weight distribution within the second fully enclosed area through a pressure sensor array to obtain a fifth item feature; and capturing the item outline and texture within the second fully enclosed area using an infrared camera within the second fully enclosed area to obtain a sixth item feature.

[0160] In one alternative approach, the fusion unit includes: calculating first evidence that the target item exists within a second fully enclosed region based on the weight similarity between the fifth item feature and the target item; calculating second evidence that the target item exists within a second fully enclosed region based on the shape similarity between the sixth item feature and the target item; and fusing the first and second evidences using an evidence theory algorithm to obtain a third matching relationship.

[0161] In one alternative embodiment, the device further includes:

[0162] The update information unit is used to receive feature update information sent by the client. The feature update information is the information fed back by the client user after confirming the target item based on the second item detection result.

[0163] The feature library update unit is used to update the feature features of items collected for the hidden area to the standard feature library in response to feature update information.

[0164] Figure 5 The diagram shows a structural schematic of an embodiment of the vehicle of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the vehicle.

[0165] The vehicle may include: processor 502, communications interface 505, memory 506, and communications bus 508.

[0166] The processor 502, communication interface 505, and memory 506 communicate with each other via communication bus 508. Communication interface 505 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps described above in the method embodiment.

[0167] Specifically, program 510 may include program code, which includes computer-executable instructions.

[0168] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The vehicle may include one or more processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0169] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0170] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0171] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0172] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0173] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0174] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for detecting items inside a vehicle, characterized in that, Applied to vehicles, the method includes: Receive a query request sent by a client, the query request being used to query a target item; In response to the query request, the vehicle's in-vehicle camera is invoked to capture images of the vehicle interior, and items in the open areas of the vehicle interior are identified through the images to obtain a first item detection result. The open areas of the vehicle interior refer to areas that can be directly observed without mechanical operation. In response to the query request, the method obtains item attribute information of a hidden area and infers a second item detection result based on the item attribute information. The item attribute information is obtained by scanning the hidden area using a hidden area sensor. The hidden area refers to a closed or semi-closed space that requires mechanical operation to access. The process of obtaining item attribute information of a hidden area in response to the query request and inferring a second item detection result based on the item attribute information includes: obtaining multiple item attribute information from multiple hidden area sensors scanning the hidden area inside the vehicle; fusing the multiple item attribute information obtained by scanning and calculating a third matching relationship between the items in the hidden area and the target item based on the fusion result; and determining the second item detection result based on the third matching relationship. The step of fusing the multiple item attribute information obtained by scanning and calculating the third matching relationship between the items in the hidden area and the target item based on the fusion result involves fusing multiple item attribute information obtained by different sensors and then calculating a third matching relationship between the items in the hidden area and the target item queried by the user based on the complete information after fusion, i.e., the similarity between the two or the credibility of the existence of the target item. The detection results of the first item and the detection results of the second item are sent to the client.

2. The method according to claim 1, characterized in that, The step of identifying items in open areas inside the vehicle through the in-vehicle image to obtain a first item detection result includes: Extract the image features of the in-vehicle image; The object detection model identifies the outlines and textures of objects in open areas inside the vehicle based on the image features. The type of item in the open area inside the vehicle is determined based on the item's outline and texture. When the outline of the item is larger than a preset outline threshold, the first matching relationship between the item type and the target item is calculated, and the detection result of the first item is determined through the first matching relationship. When the outline of the item is smaller than a preset outline threshold, the item type and the in-vehicle image are sent to the client so that the client can confirm the second matching relationship between the item type and the target item with the user. The system receives a confirmation instruction sent by the client and, in response to the confirmation instruction, outputs the first item detection result. The confirmation instruction is sent by the client when it receives information from the user confirming the second matching relationship.

3. The method according to claim 1, characterized in that, When the hidden area requested in the query is a semi-closed area, the acquisition of multiple hidden area sensors scanning multiple item attribute information in the hidden area inside the vehicle includes: The first attribute information is obtained by detecting whether an item is placed in the semi-enclosed area using a through-beam infrared sensor; And / or, locate the object's position using a beacon sensor to obtain the second attribute information; And / or, by identifying environmental information through scene sensors and estimating the types of items that may be carried based on the environmental information, third attribute information is obtained; And / or, by using an in-vehicle camera to photograph the parts of an object exposed outside the semi-enclosed area, to obtain the first object feature; And / or, by measuring the weight of the item within the semi-enclosed area using a pressure sensor array, a second item characteristic can be obtained.

4. The method according to claim 3, characterized in that, The step of fusing multiple item attribute information obtained from scanning and calculating a third matching relationship between items in the hidden area and the target item based on the fusion result includes: Based on the first attribute information, the second attribute information, and the third attribute information, respectively, we can infer the first probability, the second probability, and the third probability that the target item is located in the semi-enclosed area; Based on the Bayesian estimation method, the first probability, the second probability, and / or the third probability are fused to obtain the location confidence of the target item in the semi-enclosed area; When neither the first item feature nor the second item feature is collected, the location confidence score is used as the third matching relationship. When the first item feature and / or the second item feature are collected, the type confidence of belonging to the target item is estimated based on the first item feature and / or the second item feature, and the type confidence and the location confidence are used as the third matching relationship.

5. The method according to claim 1, characterized in that, When the hidden area queried in the query request is a first fully enclosed area, the first fully enclosed area is used to represent a frequently opened fully enclosed area. The step of obtaining multiple hidden area sensor data by scanning multiple item attribute information in the hidden areas inside the vehicle includes: The weight distribution within the first fully enclosed area is collected using a pressure sensor array to obtain the third item characteristics; When the first fully enclosed area is opened, the outline and texture of the items within the first fully enclosed area are captured by the in-vehicle camera to obtain the fourth item feature; And / or, by locating the position of the item within the first fully enclosed area using a beacon sensor, the fourth attribute information is obtained.

6. The method according to claim 5, characterized in that, The step of fusing multiple item attribute information obtained from scanning and calculating a third matching relationship between items in the hidden area and the target item based on the fusion result includes: An item feature vector is created based on the fourth attribute information, the third item feature, and the fourth item feature; Obtain the standard feature vector of the target item; The third matching relationship is obtained by calculating the distance between the item feature vector and the standard feature vector.

7. The method according to claim 1, characterized in that, When the hidden area queried in the query request is a second fully enclosed area, the second fully enclosed area represents a fully enclosed area that is not frequently opened. The step of acquiring multiple hidden area sensor data by scanning various item attribute information within the vehicle's hidden areas includes: The weight distribution within the second fully enclosed area is collected using a pressure sensor array to obtain the fifth item characteristic; The sixth item feature is obtained by capturing the outline and texture of the item within the second fully enclosed area using an infrared camera located within the second fully enclosed area.

8. The method according to claim 7, characterized in that, The step of fusing multiple item attribute information obtained from scanning and calculating a third matching relationship between items in the hidden area and the target item based on the fusion result includes: Based on the characteristics of the fifth item and the weight similarity of the target item, first evidence is calculated that the target item exists within the second fully enclosed area; Based on the characteristics of the sixth item and the similarity in appearance to the target item, second evidence is calculated that the target item exists within the second fully enclosed area; The third matching relationship is obtained by fusing the first and second pieces of evidence using evidence theory algorithms.

9. A vehicle, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the in-vehicle item detection method as claimed in any one of claims 1-8.

Images