A system and method for finding lost items in a vehicle

Abstract

The present invention discloses a system and method for finding lost items in a vehicle. The system includes a mobile phone, an authority control module, a vehicle camera, a vehicle light, and an AR presentation module. The mobile phone is used to control the opening and closing of the vehicle camera and the vehicle light through the authority control module. The vehicle camera is used to obtain images inside and outside the vehicle, and output the real-time image inside the vehicle to the AR presentation module. The AR presentation module is used to generate a real-time AR image based on the real-time image inside the vehicle, adjust the parameters of the real-time AR image according to different scenes, and transmit the adjusted real-time AR image to a mobile phone application. The system and method of the present invention interact the virtual world and the real world through real-time AR images, thereby improving the efficiency and fun of the search.

Description

Technical Field

[0001] The present invention relates to the field of AR application technology, and in particular to a system and method for finding lost items inside a vehicle. Background Art

[0002] Currently, HMI systems are based on monitoring devices or sensors inside the vehicle, as well as the display screen of the in-vehicle entertainment system, which can realize the function of finding items in the car. Users can trigger the in-vehicle monitoring device by operating the central control panel or touching the screen to find the lost items. Image recognition technology can also be used to analyze the images in the car and mark the location of possible items on the display. In addition, there are some tracking devices based on Bluetooth or Wi-Fi, which users can attach to the items and then track the location of the lost items through mobile phone applications.

[0003] The above-mentioned method for searching for items in the vehicle has the following disadvantages:

[0004] 1) Complicated user operation: Users need to manually operate the central control panel or touch the screen, which is not convenient enough and cannot intuitively find items in the car.

[0005] 2) Limitations of Bluetooth or Wi-Fi tracking devices: The positioning accuracy of tracking devices is limited by the coverage range of Bluetooth or Wi-Fi signals. If the item moves to an uncontrollable range, it will not be able to be located. Summary of the invention

[0006] In view of the deficiencies of the lost item search function in the prior art, the present invention provides a system and method for finding lost items in a vehicle, which can improve the efficiency and interest of searching for lost items.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0008] The present invention is a system for finding lost items in a vehicle, the system comprising a mobile phone, an authority control module, a vehicle camera, a vehicle light, and an AR presentation module;

[0009] The mobile phone is used to control the on-board camera and the on-board light to be turned on and off through the authority control module;

[0010] The vehicle camera is used to obtain images inside and outside the vehicle, and output real-time images inside the vehicle to the AR presentation module;

[0011] The AR presentation module is used to generate a real-time AR image based on the real-time image inside the car, adjust the parameters of the real-time AR image according to different scenes, and transmit the adjusted real-time AR image to the mobile phone application.

[0012] A further improvement of the present invention is that: the system further comprises a target image uploading and processing module and a target image marking module;

[0013] The target image uploading and processing module is used to upload the lost item image as the target image, analyze and process the lost item image, and extract image features;

[0014] The target image marking module is used to perform image feature recognition on the target image in the real-time image output by the AR presentation module, and mark the matching image in the real-time image.

[0015] A further improvement of the present invention is that the target image uploading and processing module processes the lost item image as follows:

[0016] Use object detection algorithms to identify target objects in lost item images;

[0017] Use image segmentation algorithm to segment the identified target object;

[0018] The feature extraction algorithm is used to extract the features of the target object from the segmented image, the extracted target object features are stored, and the extracted target object features are input into the target image marking module.

[0019] A further improvement of the present invention is that the target image marking module performs the following operations: using a trained deep learning model to perform feature matching operations between the real-time AR image and the target object. If the match is successful, the label and location information of the target object are obtained, and a label is generated in the real-time AR image based on the label and location information of the target object.

[0020] A further improvement of the present invention is that: the system further includes a cloud server and a preset image processing unit;

[0021] The preset image processing unit is used to store the preset image uploaded by the mobile phone and analyze and process the preset image;

[0022] The cloud server is used to store and analyze image information acquired by the vehicle camera, and to exchange information with the mobile phone through the communication module.

[0023] A further improvement of the present invention is that the parameters of the real-time AR image include transparency parameters, scaling parameters, and lighting and shadow parameters, and the transparency parameters, scaling parameters, and lighting and shadow parameters are adjusted by an adaptive algorithm.

[0024] The present invention is a method for finding lost items in a vehicle, comprising the following steps:

[0025] Get real-time AR images of the vehicle interior;

[0026] Lost items are searched through real-time AR images. If the lost items are not found, image recognition technology is used to match the target image with the video data obtained by the vehicle camera, and the video segment containing the target image is extracted. The lost items are tracked and searched through the video segment.

[0027] A further improvement of the present invention is that the specific steps of obtaining the real-time AR image inside the vehicle are:

[0028] Step a: The user requests permission to control the vehicle camera and vehicle lights, including:

[0029] Step a1, the user opens the mobile phone application for AR to find lost items;

[0030] Step a2, the mobile phone application checks the mobile phone permission status to see whether it has the permission to control the vehicle camera and vehicle lights;

[0031] Step a3: If the permission is not enabled, the mobile app displays a prompt to the user requesting control permission and waits for the user to authorize.

[0032] Step b: After the user obtains the permission to control the vehicle camera and vehicle lights, he turns on the vehicle lights and vehicle camera through the mobile phone;

[0033] In step c, the vehicle camera obtains the real-time image inside the vehicle and transmits the real-time image inside the vehicle to the AR rendering module. The AR rendering module generates a real-time AR image and transmits it to the mobile phone application.

[0034] A further improvement of the present invention is that the method of searching for lost items through real-time AR images specifically includes any one or two of the following methods:

[0035] Method 1:

[0036] Users can use the mobile app to view real-time AR images inside the car and find lost items;

[0037] Method 2:

[0038] The user uploads the image of the lost item to the mobile application, and the target object in the lost item image is identified using the object detection algorithm;

[0039] Use image segmentation algorithm to segment the identified target object;

[0040] The feature extraction algorithm is used to extract the features of the target object from the segmented image, and the extracted target object features are stored. The extracted target object features are input into the trained deep learning model to match the real-time AR image with the target object features. If the match is successful, the label and location information of the target object are obtained, and a marker is generated in the real-time AR image based on the label and location information of the target object.

[0041] A further improvement of the present invention is that, in the case where the lost item is not found, a preset image is uploaded, which includes an image of the lost item and a target image related to the lost item. Multiple targets including the lost item in the preset image are identified through image recognition technology, and target features of the multiple targets are extracted. Target feature recognition is performed on the video acquired through the vehicle camera through a trained image recognition model. If the recognition is successful, a video segment containing multiple targets is extracted, and the user tracks and searches for the lost item through the extracted video segment.

[0042] The beneficial effects of the present invention are as follows: the present invention applies AR technology to a lost item search system in a car, provides a new search method, can display virtual information of the car or surrounding environment in real time on the mobile phone screen, and users can search more intuitively through visual perception, avoiding time-consuming operations such as rummaging. The present invention provides a variety of search methods, using AR technology to find lost items. On the basis of improving the fun of searching for lost items, users can also choose to mark lost items in real-time AR images to quickly find lost items; in addition, if the lost items are not found through the above two methods, users can also choose to upload images of lost items and target images related to the lost items, and retrieve historical video data stored in the car camera through image recognition technology, so as to track and find lost items based on historical video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of a system for finding lost items in a vehicle in an embodiment of the present invention;

[0044] Figure 2 The present invention is a flowchart of a method for finding lost items in a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0046] like Figure 1As shown, a system for finding lost items in a vehicle according to this embodiment includes a mobile phone, an authority control module, a vehicle camera, a vehicle light, and an AR presentation module;

[0047] The mobile phone is used to control the on-board camera and the on-board light to be turned on and off through the authority control module;

[0048] The vehicle camera is used to obtain images inside and outside the vehicle and output real-time images inside the vehicle to the AR presentation module;

[0049] The AR presentation module is used to generate a real-time AR image based on the real-time image inside the car, use an adaptive algorithm to adjust the parameters of the real-time AR image according to different scenes, and send the adjusted real-time AR image to the mobile phone application.

[0050] The above modules cooperate with each other in the whole system, and realize the function of AR finding lost items through the interaction of attributes, events and methods. The specific operation steps are as follows:

[0051] Step 1: The user requests permission to control the vehicle camera and vehicle lights, including:

[0052] Step 11, the user opens the mobile phone application for AR to find lost items;

[0053] Step 12: The mobile phone application checks the mobile phone permission status to see whether it has the permission to control the vehicle camera and vehicle lights;

[0054] Step 13: If the permission is not enabled, the mobile app displays a prompt to the user requesting control permission and waits for the user to authorize;

[0055] Step 2: After the user obtains the permission to control the vehicle camera and vehicle lights, he turns on the vehicle lights and vehicle camera through the mobile phone;

[0056] Step 3: The vehicle camera obtains the real-time image inside the vehicle and transmits the real-time image inside the vehicle to the AR rendering module. The AR rendering module generates a real-time AR image and transmits it to the mobile phone application;

[0057] Step 4: The user uses the mobile phone application to view the real-time AR image inside the car to find the lost item;

[0058] When not driving the vehicle, the user can request to turn off the vehicle camera and lights through the mobile phone application after finding the lost item or no longer needing the real-time AR image guidance in the vehicle. After receiving the user's request, the permission control module controls the vehicle camera and lights to turn off.

[0059] The mobile phone is used as a control button to control the car camera and lights, and obtain real-time images inside the car through the car camera. The AR presentation module combines the real-time images inside the car to output real-time AR images inside the car. Users can use the real-time AR images inside the car to find lost items in the car.

[0060] Another embodiment of the present invention further includes a target image uploading and processing module and a target image marking module based on the above embodiment;

[0061] The target image uploading and processing module is used to upload the lost item image as the target image, analyze and process the lost item image, and extract image features;

[0062] The target image marking module is used to perform image feature recognition on the target image in the real-time image output by the AR presentation module, and mark the matching image in the real-time image.

[0063] The specific operations for analyzing and processing lost items images are as follows:

[0064] Use object detection algorithms to identify target objects in lost item images;

[0065] Use image segmentation algorithm to segment the identified target object;

[0066] The feature extraction algorithm is used to extract the features of the target object from the segmented image, the extracted target object features are stored, and the extracted target object features are input into the target image marking module.

[0067] The target image labeling module performs the following operations: Use the trained deep learning model to perform feature matching operations between the real-time AR image and the target object. If the match is successful, the label and location information of the target object are obtained, and a label is generated in the real-time AR image based on the label and location information of the target object.

[0068] If the user does not find the lost item through the real-time AR image or wants to quickly locate the lost item, the user can choose to upload the lost item image, extract the features of the lost item image, and use the target image marking module to mark the lost item in the real-time AR image. Marking the lost item in the real-time AR image can help users quickly find the lost item.

[0069] The system of another embodiment of the present invention further includes a cloud server and a preset image processing unit;

[0070] The preset image processing unit is used to store the preset images uploaded by the mobile phone and analyze and process the preset images, wherein the preset images include the lost item image and the target image related to the lost item. The target image related to the lost item can be an image of a related item lost together with the lost item, or a face image of the user or passenger who lost the lost item;

[0071] The cloud server is used to store and analyze image information acquired by the vehicle camera, and to exchange information with the mobile phone through the communication module.

[0072] If the user has not found the lost item, he can upload a preset image, use image recognition technology to identify multiple targets including the lost item in the preset image, extract the target features of multiple targets, and use the trained image recognition model to identify the target features of the video obtained by the car camera. If the recognition is successful, the video segment containing multiple targets is extracted, and the user tracks and searches for the lost item through the extracted video segment. For example: After the user finds that the item is lost in the car, he can search for the lost item through the real-time AR image. If he has not found it, he can upload the image of the lost item and mark the lost item in the real-time AR image. If no marks appear in the real-time AR image, if the user remembers the last person who took out the lost item in the car, the user can upload pictures of the lost item and the face pictures of the people related to the lost item, and use image recognition technology to identify and extract the features of the lost item and the face features, and use the trained image recognition model to perform target feature recognition on the historical videos obtained through the car camera, and extract video segments of the lost item and the people related to the lost item. The user can combine the approximate time when the lost item was last taken out of the car and select the video of that time period to view the location where the lost item was lost.

[0073] The parameters of the real-time AR image in the present invention include transparency parameters, scaling parameters, and lighting and shadow parameters. The transparency parameters, scaling parameters, and lighting and shadow parameters are adjusted by an adaptive algorithm to help users locate lost items faster. Among them, the transparency parameter controls the visibility of virtual information. When the transparency is high, the virtual information will be superimposed on the real image semi-transparently, and when the transparency is low, the virtual information will be fully displayed, covering part or all of the real image. The current light is obtained by the light sensor, so as to adjust the transparency parameters of the AR presentation, so as to achieve a smooth transition between the virtual information and the real scene. The light sensor in the mobile phone can monitor the light intensity in the car in real time, in units of lumens (lm). The light sensor reads the light intensity in the car regularly (30s) and converts it into lumens (lm). Assume that the current light intensity in the car is 500 lumens. The AR presentation module uses the obtained light intensity to calculate the transparency parameter. A transparency function can be set to map the light intensity to the range of the transparency parameter (between 0 and 1). For example, a linear mapping can be used: transparency parameter = light intensity / maximum light intensity. Assuming the maximum light intensity is 1000 lumens, the current transparency parameter is: transparency parameter = 500 / 1000 = 0.5. The AR rendering module uses the calculated transparency parameter to present virtual information, namely the marker arrow. In this example, if the transparency parameter is 0.5, the marker arrow will be displayed semi-transparently in the real scene, and the user can see the real scene and virtual information through the marker arrow. As the light intensity in the car changes, the light sensor will continuously update the acquired light information. The AR rendering module can recalculate the transparency parameter according to the new light intensity to achieve a smooth transition between virtual information and the real scene. For example, if the light intensity increases to 700 lumens, the new transparency parameter is: transparency parameter = 700 / 1000 = 0.7. This will cause the marker arrow to become more transparent in the real scene to adapt to the brighter lighting environment. The specific transparency parameter can be set to:

[0074] Low Light: 0.2, to enhance the visibility of the marker arrow.

[0075] Medium Light: 0.5, makes the marker arrow stand out in relatively bright environments.

[0076] Highlight: 0.8, making the marker arrow semi-transparent in bright environments without blocking the view too much.

[0077] Scaling parameter: adjust the size of virtual information in the real-time AR image. Through the scaling parameter, the size of the virtual information can be controlled to match the size of objects in the real scene and enhance the visual fusion effect. The scaling parameter is a proportional factor used to enlarge or reduce the size of the virtual information. If the scaling parameter is set to 1.0, it means that the virtual information is displayed at the original size; if the scaling parameter is set to 0.5, it means that the size of the virtual information in the real-time AR image will be half of the original size. By increasing or decreasing the scaling parameter, the virtual information can adapt to the size of objects in the real scene, so that the virtual information is closer to the real scene, enhancing the realism and immersion of the virtual information. In some scenarios, it may be necessary to scale the virtual information to completely match the real object (such as the information description of the item) to achieve a more realistic AR experience. In other scenarios, in order to emphasize the importance of the virtual information or increase its visibility, it may be scaled to not completely match the real object, making it more conspicuous in the AR scene (such as directional instructions). The scaling parameter can be automatically adjusted dynamically according to the current page operated by the user. The following is a specific example of the scaling parameter of this embodiment:

[0078] Back row: 0.7, so that the marking arrows of virtual information are appropriately reduced in the spacious space.

[0079] Corner: 1.3, Increase the size of the marker arrows in limited space to be more visible.

[0080] Seat: 1.0. Keep the arrow marking the virtual information consistent with the product size.

[0081] Whole vehicle: 0.9, slightly reduce the size of the marking arrow to adapt to the outdoor environment.

[0082] Lighting and shadow parameters: According to the lighting conditions in the car, the AR rendering module can calculate the shadow of the virtual information according to the lighting parameters and project the shadow into the real world, so that the virtual information presents a natural shadow effect in the car. Assume that in the coordinate system in the car, with the center point as the origin, the position of the roof light is (0, 3, 0), and the light intensity is 1000 lumens. The surface position of the seat is (1, 0.5, 0). Light source detection, the position of the roof light is known, and the light intensity is 1000 lumens. Shadow projection, assuming that the direction of the roof light is along the negative Y-axis direction, the intersection position of the roof light and the seat surface can be obtained as (1, 2.5, 0), which indicates the position where the light shines from the roof light to the seat surface. Light occlusion, assuming that there are no other objects in the car to block the light, the light can directly shine on the seat surface. Shadow rendering, assuming that the intensity of light decays by 50% after a certain distance, that is, the regional illumination intensity of the light irradiating the seat surface is 1000 * 0.5 = 500 lumens. According to the light intensity, the brightness and gradient effect of the shadow on the seat surface can be calculated. Virtual information superposition, finally, the calculated shadow effect is superimposed with the virtual information (such as the mobile phone mark arrow). If the color of the virtual information on the seat surface is red, then considering the shadow effect, the color of the virtual information on the seat surface may become slightly darker, such as dark red. For example, the shadow parameters of the illumination can be set as:

[0083] Dusk: 0.7 to simulate the light effect at dusk.

[0084] Daytime Bright: 0.3, to accommodate strong sunlight and bright lighting conditions.

[0085] Natural sunlight: 0.5, simulates the effect of natural lighting.

[0086] Dark environment: 0.9, simulating very dim light conditions.

[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A system for finding lost items in a vehicle, characterized in that: The system includes a mobile phone, an authority control module, a vehicle camera, vehicle lights, and an AR presentation module; The mobile phone is used to control the on-board camera and the on-board light to be turned on and off through the authority control module; The vehicle camera is used to obtain images inside and outside the vehicle, and output real-time images inside the vehicle to the AR presentation module; The AR presentation module is used to generate a real-time AR image based on the real-time image in the car, adjust the parameters of the real-time AR image according to different scenes, and send the adjusted real-time AR image to the mobile phone application; The system also includes a target image uploading and processing module and a target image marking module; The target image uploading and processing module is used to upload the lost item image as the target image, analyze and process the lost item image, and extract image features; The target image marking module is used to perform image feature recognition on the target image in the real-time image output by the AR rendering module, and mark the matching image in the real-time image; The target image uploading and processing module processes the lost item image as follows: Use object detection algorithms to identify target objects in lost item images; Use image segmentation algorithm to segment the identified target object; Extracting features of the target object from the segmented image using a feature extraction algorithm, storing the extracted features of the target object, and inputting the extracted features of the target object into a target image marking module; The target image marking module performs the following operations: using the trained deep learning model to perform a feature matching operation between the real-time AR image and the target object, if the match is successful, obtaining the label and location information of the target object, and generating a label in the real-time AR image according to the label and location information of the target object; The parameters of the real-time AR image include transparency parameters, scaling parameters, and lighting and shadow parameters, and the transparency parameters, scaling parameters, and lighting and shadow parameters are adjusted through an adaptive algorithm.

2. A lost item search system in a vehicle according to claim 1, characterized in that: The system also includes a cloud server and a preset image processing unit; The preset image processing unit is used to store the preset image uploaded by the mobile phone and analyze and process the preset image; The cloud server is used to store and analyze image information acquired by the vehicle camera, and to exchange information with the mobile phone through the communication module.

3. A method for finding lost items in a vehicle, characterized in that: The steps include: Get real-time AR images of the vehicle interior; Search for lost items through real-time AR images. If the lost items are not found, image recognition technology is used to match the target image with the video data obtained by the vehicle camera, and the video segment containing the target image is extracted to track and search for the lost items through the video segment; The specific steps of obtaining the real-time AR image inside the vehicle are: Step a: The user requests permission to control the vehicle camera and vehicle lights, including: Step a1, the user opens the mobile phone application for AR to find lost items; Step a2, the mobile phone application checks the mobile phone permission status to see whether it has the permission to control the vehicle camera and vehicle lights; Step a3: If the permission is not enabled, the mobile application displays a prompt to the user requesting control permission and waits for the user to authorize. Step b: After the user obtains the permission to control the vehicle camera and vehicle lights, he turns on the vehicle lights and vehicle camera through the mobile phone; Step c: The vehicle camera obtains the real-time image inside the vehicle and transmits the real-time image inside the vehicle to the AR rendering module. The AR rendering module generates a real-time AR image and transmits it to the mobile phone application; The method of finding lost items through real-time AR images specifically includes any one or two of the following methods: Method 1: Users can use the mobile app to view real-time AR images inside the car and find lost items; Method 2: The user uploads the image of the lost item to the mobile application, and the target object in the lost item image is identified using the object detection algorithm; Use image segmentation algorithm to segment the identified target object; The feature extraction algorithm is used to extract the features of the target object from the segmented image, and the extracted target object features are stored. The extracted target object features are input into the trained deep learning model to match the real-time AR image with the target object features. If the match is successful, the label and location information of the target object are obtained, and a marker is generated in the real-time AR image based on the label and location information of the target object.

4. A method for finding lost items in a vehicle according to claim 3, characterized in that: If the lost item is not found, a preset image is uploaded, which includes an image of the lost item and a target image related to the lost item. Multiple targets including the lost item in the preset image are identified through image recognition technology, and target features of the multiple targets are extracted. The target features are then identified in the video acquired through the vehicle camera through a trained image recognition model. If the identification is successful, the video segment containing multiple targets is extracted, and the user can track and search for the lost item through the extracted video segment.

Images