Mobile phone forgetting reminding method and device, electronic equipment and vehicle

Abstract

The invention provides a mobile phone forgetting reminding method and device, electronic equipment and a vehicle, and relates to the field of image recognition. The method comprises the following steps of: identifying handheld articles of passengers by using a vehicle-mounted camera; when the handheld object is a mobile phone, tracking the track of the mobile phone; when the mobile phone is separated from the passenger, marking the mobile phone as a separated state; and finally, in response to a vehicle leaving signal of the passenger, sending a reminding signal to the passenger according to the mobile phone in the separated state. According to the technical scheme of the embodiment, the mobile phone of the passenger is locked, the track of the mobile phone is monitored, and the mobile phone is marked when the mobile phone is not in contact with the passenger, so that when the passenger leaves the vehicle, it is confirmed that the mobile phone is forgotten in the vehicle by the passenger, the reminding signal is sent to the passenger, and the passenger is reminded to take the passenger. The situation that a passenger needs to turn back and take back again due to the fact that the mobile phone is forgotten in the car can be prevented, and time waste of the passenger is reduced.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and in particular to a method, device, electronic equipment and vehicle for reminding a person to forget a mobile phone. Background Art

[0002] Passengers often use their mobile phones while driving. After using their mobile phones, they will put them in different places in the car. After the vehicle arrives at the destination, the passengers may forget that their mobile phones are still in the vehicle due to various reasons, and need to return to the vehicle to retrieve the mobile phones, which wastes time. Summary of the invention

[0003] In view of this, the present application provides a forgotten mobile phone reminder method, device, electronic device and vehicle, the main purpose of which is to remind passengers that there is a forgotten mobile phone in the vehicle when the passengers leave the vehicle, thereby reminding the passengers to retrieve the forgotten mobile phone in the car, preventing the passengers from returning to the vehicle to retrieve the mobile phone at a long distance away from the vehicle, and reducing the passengers' time waste.

[0004] To achieve the above-mentioned purpose, the first aspect of the present application discloses a method for reminding a forgotten mobile phone, the method comprising:

[0005] Use the vehicle's camera to identify the passenger's handheld items;

[0006] When the handheld object is a mobile phone, tracking the trajectory of the mobile phone;

[0007] When the mobile phone is separated from the occupant, adding a separation mark to the mobile phone;

[0008] In response to a vehicle leaving signal from the occupant, a reminder signal is sent to the occupant according to the separation mark.

[0009] Optionally, the identifying of the passenger's handheld items by using a vehicle-mounted camera includes:

[0010] Using a vehicle-mounted camera to capture an image including the occupant, and identifying the content of the image;

[0011] Scanning the shape and size of the handheld object, and using the shape and size of the handheld object as first weight information;

[0012] identifying the occupant's control action on the handheld object, and using the control action information as second weight information;

[0013] The first weight information and the second weight information are combined to obtain a recognition result of the handheld object.

[0014] Optionally, the identifying the occupant's control action on the handheld object to obtain second weight information includes:

[0015] Identifying the relative direction of the occupant's face and the handheld object to obtain direction information;

[0016] identifying the occupant's operation action on the handheld object to obtain action information;

[0017] Calculating the duration of the operation action to obtain time information;

[0018] The control action information is generated by combining the direction information, the action information and the time information, and the control action information is used as the second weight information.

[0019] Optionally, when the mobile phone is separated from the occupant, after adding a separation mark to the mobile phone, the method further includes:

[0020] When the occupant comes into contact with the mobile phone again, canceling the separation state mark of the mobile phone;

[0021] The trajectory of the mobile phone is tracked again.

[0022] Optionally, in response to the occupant's vehicle leaving signal, sending a reminder signal to the occupant according to the separation mark includes:

[0023] In response to the occupant's leaving signal, waking up the vehicle's signal display device according to the separation mark, the signal display device comprising: at least one of a vehicle-mounted display screen, a warning light, and a speaker;

[0024] The signal device is controlled to send a reminder signal to the occupant according to a preset reminder strategy.

[0025] Optionally, the controlling the signal device to send a reminder signal to the occupant according to a preset reminder strategy includes:

[0026] When the signal display device includes the vehicle-mounted display screen, controlling the vehicle-mounted camera to identify the location of the mobile phone;

[0027] The location of the mobile phone is displayed via the vehicle-mounted display screen.

[0028] Optionally, after sending a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal, the method further includes:

[0029] In response to a cancellation signal of the separation status mark or a confirmation signal of the occupant that the occupant has not forgotten, the reminder signal is turned off.

[0030] In a second aspect of the present application, an embodiment provides a forgotten mobile phone reminder device, the device comprising:

[0031] An identification module is used to identify objects held by passengers using an on-board camera;

[0032] A tracking module, used for tracking the trajectory of the mobile phone when the handheld object is a mobile phone;

[0033] a marking module, used for adding a separation mark to the mobile phone when the mobile phone is separated from the occupant;

[0034] The reminder module is used to respond to the occupant's leaving vehicle signal and send a reminder signal to the occupant according to the mobile phone in the separated state.

[0035] In a third aspect of the present application, an electronic device is provided, including:

[0036] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods disclosed in the first aspect.

[0037] In a fourth aspect of the present application, an embodiment provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0038] In a fifth aspect of the present application, a vehicle is provided, in which the forgotten mobile phone reminder device as described in the second aspect or the electronic device as described in the third aspect is mounted.

[0039] In summary, according to the technical solution disclosed in the present application, in order to prevent passengers from forgetting their mobile phones in the car when leaving the car, which causes the passengers to spend more time to return to the car to retrieve the mobile phones, resulting in unnecessary waste of time. The present application discloses a method for reminding passengers of forgotten mobile phones. Through the technical solution of this embodiment, visual recognition technology is used to lock the passenger's mobile phone, monitor the trajectory of the mobile phone, and mark the mobile phone when it is not in contact with the passenger. When the passenger leaves the car, it is confirmed that the mobile phone is forgotten in the car by the passenger, and a reminder signal is sent to the passenger. By adopting the technical solution of this implementation, visual recognition technology can be used to confirm the separation relationship between the mobile phone and the passenger, and a reminder signal is sent when the passenger leaves the car, which can prevent the passenger from having to return to retrieve the mobile phone because it is forgotten in the car, thereby reducing the passenger's time waste.

[0040] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of a forgotten mobile phone reminder method provided by an embodiment of the present application is shown;

[0044] Figure 2 A structural diagram of a forgotten mobile phone reminder device provided in an embodiment of the present application is shown.

[0045] Figure 3 A structural diagram of a forgotten mobile phone reminder operation system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0047] In order to solve the problem that when a vehicle is driving, passengers place their mobile phones randomly after using them and forget to retrieve them when leaving the vehicle, which wastes time in the subsequent retrieval process. The present application provides the following embodiments to solve the above problem:

[0048] This embodiment provides a method for reminding a forgotten mobile phone. Figure 1 FIG. 1 is a flowchart of the method of this embodiment, and the method of this embodiment may specifically include the following steps:

[0049] Step 101, using the vehicle-mounted camera to identify the passenger's handheld items.

[0050] In the present embodiment, the vehicle camera first confirms the occupants in the vehicle cabin, and then confirms the hand movements of the user. When confirming that the user's hand holds an object, the type of the handheld object is further identified. Further, the handheld object is identified as a mobile phone. In the process of identifying the handheld object, whether it is a mobile phone can be accurately identified based on the user's use of the mobile phone.

[0051] Regarding the identification method of handheld objects, the present application proposes a specific embodiment, which uses a vehicle-mounted camera to identify the handheld objects of the occupants, including:

[0052] An on-board camera is used to capture images including passengers and the contents of the images are identified; the shape and size of a handheld object are scanned and the shape and size of the handheld object are used as first weight information; the passenger's control action on the handheld object is identified and the control action information is used as second weight information; the first weight information and the second weight information are combined to obtain an identification result of the handheld object.

[0053] In the content of this embodiment, in order to confirm that the user's handheld item is a mobile phone, this specific embodiment discloses a method for identifying handheld items. First, an image containing an occupant taken by a camera is obtained, and the image content is further identified. When the occupant's hand holding item is determined by the recognition result of the image, the detailed size of the handheld item is confirmed based on the reference object. It can be understood that the specific size of the mobile phones on the current market is adjusted within a certain preset range in order to adapt to the user's two-handed operation. At the same time, most mobile phones are rectangular or rounded rectangles. Therefore, the size of the handheld item can be confirmed by identifying the length and width of the handheld item, as the first weight information, to determine whether it is a mobile phone. For example, when 7*15cm is identified as a rectangle, the weight W1 given to the rectangle as a mobile phone is 0.7.

[0054] After determining the shape and size, considering that the existing mobile phone may be displayed as an irregular shape in the captured image due to the external mobile phone protective case, thus causing the unrecognizable situation, this embodiment further proposes a process of calculating the core area of ​​the handheld object. Although the outline of the mobile phone is affected by the size and shape of the mobile phone protective case, it is impossible to directly and accurately confirm that the object is a mobile phone, but the core area of ​​the mobile phone with the external mobile phone protective case remains unchanged, and it can also be determined whether it meets the size of the mobile phone by identifying the core area.

[0055] In the content of this embodiment, in order to further confirm whether the handheld object is a mobile phone, it is also proposed to judge the handheld object according to the user's control action on the handheld object. It should be understood that in order to support the user's use of the mobile phone, the mobile phone supports executing the function corresponding to the action when the user performs various control actions. Therefore, the content of this embodiment further scans and identifies the user's control actions, for example, raising the mobile phone with both hands to take a photo, continuously pressing the screen to send a voice, etc. The above-mentioned raising both hands and continuously pressing the screen can be used as control actions, and the control action information of the user's control action is extracted, and the second weight information is confirmed by extracting the control action information.

[0056] Based on this embodiment, it is proposed to combine the first weight information and the second weight information to jointly identify whether the handheld item of the occupant is a mobile phone. The final weight result is calculated based on the first weight information and the second weight information, and whether the handheld result is a mobile phone is confirmed based on the final weight result. Through the visual recognition of the vehicle-mounted camera, the first weight information and the second weight information are used to jointly confirm whether it is a mobile phone. The accuracy of identifying the user's handheld item as a mobile phone through visual recognition is guaranteed, and subsequent tracking and reminders are achieved on this basis.

[0057] In a possible embodiment, with respect to the process of obtaining the second weight information, the present application further provides an embodiment content, which identifies the control action of the occupant on the handheld object and obtains the second weight information, including:

[0058] Identify the relative direction between the occupant's facial orientation and the handheld item to obtain direction information; identify the occupant's operating actions on the handheld item to obtain action information; calculate the duration of the operating action to obtain time information; combine the direction information, action information and time information to generate control action information, and use the control action information as the second weight information.

[0059] In the process of obtaining the second weight information, the user's control action on the handheld object is used to determine whether the handheld object is a mobile phone. This embodiment discloses a more specific control action recognition process.

[0060] Specifically, first identify the relative direction between the user's face and the handheld item. It is known that when the handheld item is a mobile phone, the main functions of the mobile phone include making and receiving calls, online instant messaging or entertainment activities. While using the above-mentioned different functions, different control actions are maintained between the user and the mobile phone. Specifically, for example, when making a call in the handset mode, the side of the occupant faces the handheld item, that is, the side face faces the handheld item; while making a call in the speaker mode, the occupant's mouth faces the handheld item, that is, the lower face faces the handheld item; and in other instant messaging methods, the occupant's eyes face the handheld item, that is, the upper face faces the handheld item. According to different usage scenarios, the relative direction of the occupant's face and the handheld item is obtained as direction information.

[0061] Secondly, identify the occupant's operating actions on the handheld items. In the process of using a mobile phone, in order to adapt to multiple functions, different operating actions need to be adopted, such as online chatting, browsing the web, etc. For example, operating actions such as continuously clicking on the handheld item or sliding the handheld item in various directions are used as action information.

[0062] Finally, the duration of the operation is calculated. It can be known that whether it is a voice call or entertainment activity through a mobile phone, it is a continuous process, so there is a duration, which can be used as an information element of the second weight information.

[0063] Direction information, motion information and time information are combined as the second weight information to calculate the recognition result of the handheld object. At the same time, different combinations of direction information, motion information and time information can calculate different control actions of the user. According to different control actions, the way the passenger uses the mobile phone can be determined and different weight information can be assigned. For example:

[0064] 1. When the occupant's upper face is facing the handheld item, and the finger slides in the handheld item area for more than 10 seconds, the control action is regarded as the browsing mode and the weight W2 is assigned to 0.6. It is generally used for browsing web pages, reading e-books, etc.

[0065] 2. If it is recognized that the occupant's upper face is facing the handheld item, and the fingers are holding the handheld item for more than 10 seconds, the control action is regarded as the browsing mode and the weight W3 is assigned to 0.6, which is generally used for watching videos, etc.

[0066] 3. Recognize that the occupant's upper face is facing the handheld object, the fingers are holding the handheld object, and the face and the arm maintain the same adjustment direction, that is, when the arm is lowered, the face is lowered; when the arm is raised, the face is raised, and the duration is more than 10 seconds respectively. This control action is used as the adjustment mode and the weight W4 is assigned to 0.6. It is generally used for occupant movement adjustment.

[0067] 4. If it is recognized that the passenger's fingers are holding a handheld item with the side of the face facing the handheld item and the duration exceeds 10 seconds, the control action is regarded as the handset call mode and the weight W5 is assigned to 0.6, which is generally used for making and receiving calls.

[0068] 5. If it is recognized that the passenger's fingers are holding a handheld item, and the lower face is facing and close to the handheld item, and the duration exceeds 10 seconds, the control action is regarded as a speaker call mode and the weight W6 is assigned to 0.6.

[0069] The above-mentioned weights W2, W3, W4, W5 or W6 are used as the second weight information, combined with the first weight information W1, to obtain the recognition result of the handheld object. The calculation process of the final weight W is:

[0070] W=W1*W2*W3*W4*W5*W6 (1)

[0071] According to the user's usage pattern of the handheld object, W1 to W6 are assigned values, and whether the handheld object is a mobile phone is determined based on the final W value.

[0072] Step 102: When the handheld object is a mobile phone, the trajectory of the mobile phone is tracked.

[0073] In this embodiment, when it is determined that the handheld object is a mobile phone, the vehicle camera is used to track the movement trajectory of the mobile phone. While the vehicle camera is used to shoot the interior of the vehicle cabin, the position of the mobile phone is continuously identified by the image generated by the continuous shooting of the vehicle camera, so as to track the trajectory of the mobile phone.

[0074] Furthermore, due to the complexity of the in-vehicle environment, a single vehicle camera is often unable to track the mobile phones of passengers at different locations in the vehicle. Therefore, the present embodiment further proposes to use the vehicle camera corresponding to each row of seats in the vehicle to track the movement trajectory of the mobile phones of passengers in that row, so as to prevent the problem of tracking failure caused by the blind spot of the camera's field of view.

[0075] Step 103: When the mobile phone is separated from the passenger, a separation mark is added to the mobile phone.

[0076] When passengers are in a vehicle, after using their mobile phones, they are accustomed to placing their mobile phones in different places in the vehicle. Commonly used places include: left and right door storage space, near the armrest box, center console, storage space on the back of the front seat facing the second row, and possibly on each row of seats. When the mobile phone is no longer in contact with the passenger's body, whether the mobile phone is placed by the passenger actively or passively slipped off the passenger, the mobile phone is separated from the passenger, and a separation mark is added to the mobile phone in the image captured by the vehicle camera.

[0077] Furthermore, an ownership relationship is established between the mobile phone with the separation mark and the occupant, that is, it is confirmed that the mobile phone with the separation mark belongs to the occupant who was separated from the mobile phone most recently.

[0078] Step 104 : in response to the occupant's vehicle leaving signal, sending a reminder signal to the occupant according to the separation mark.

[0079] When a signal indicating that a passenger has left the vehicle is confirmed, for example, when a sensor such as a gravity sensor or an external camera detects that the passenger has left the vehicle and is moving away from the vehicle, if the image of the vehicle interior captured by the onboard camera contains a mobile phone with a separation mark, a reminder signal is sent to the passenger to make the passenger aware of the presence of forgotten items in the vehicle. This prevents the occupants from mistakenly sending a reminder signal to the passenger who got off the vehicle first because of a mobile phone in a separated state belonging to the passenger who got off the vehicle later when the passengers get off the vehicle in batches.

[0080] Furthermore, according to the correspondence between the mobile phone with the separation mark and the occupant, when the occupant leaves the vehicle, the reminder device of the vehicle is controlled to send a reminder signal to the occupant getting off the vehicle to remind him / her of the forgotten items.

[0081] In the above embodiment, the vehicle camera is used to identify the handheld items of the occupant first; then when the handheld item is a mobile phone, the trajectory of the mobile phone is tracked; then when the mobile phone is separated from the occupant, a separation mark is added to the mobile phone; finally, in response to the occupant's leaving signal, a reminder signal is sent to the occupant according to the separation mark. The technical solution of this embodiment is adopted. During the occupant's riding process, the vehicle camera is used to shoot and identify the handheld items of the occupant, and accurately identify them. When it is determined that the handheld item is a mobile phone, the movement trajectory of the mobile phone is tracked, the state of the mobile phone being separated from the occupant is captured, and when the occupant leaves the car, a reminder signal is initiated according to the mobile phone with the separation mark. Through the technical solution of this embodiment, the visual recognition technology is used to lock the occupant's mobile phone, monitor the trajectory of the mobile phone, and mark the mobile phone when it is not in contact with the occupant, so that when the occupant leaves the car, it is confirmed that the mobile phone is forgotten in the car by the occupant, and a reminder signal is sent to the occupant. By adopting the technical solution implemented in this embodiment, visual recognition technology can be used to confirm the separation relationship between the mobile phone and the occupant, and a reminder signal can be sent when the occupant leaves the car. This can prevent the occupant from having to return to the car to retrieve the mobile phone due to forgetting it in the car, thereby reducing the occupant's time waste.

[0082] At the same time, in the above-mentioned process of confirming that the occupant's handheld item is a mobile phone, a calculation method combining the conventional size and conventional usage of the mobile phone is used to further determine that the handheld item is a mobile phone. Combined with the recognition of the image taken by the vehicle-mounted camera, it provides an accurate basis for the visual recognition system of the vehicle-mounted system to accurately identify the mobile phone, thereby ensuring the accuracy of the vehicle-mounted system in identifying the mobile phone, further ensuring the effectiveness of subsequent actions on the mobile phone, and ensuring the reliability of this embodiment during implementation.

[0083] In a possible embodiment, when the mobile phone is separated from the passenger, after adding a separation mark to the mobile phone, the method further includes:

[0084] When the occupant comes into contact with the mobile phone again, the separation mark of the mobile phone is cancelled; and the trajectory of the mobile phone is tracked again.

[0085] In this embodiment, when the mobile phone is separated from the passenger, a separation mark is added to the mobile phone. However, no matter whether the passenger is separated from the mobile phone actively or passively, the passenger can take the mobile phone back. When the vehicle camera captures the mobile phone and the passenger's re-contact, it is determined that the mobile phone is no longer in a separated state, the separation mark is cancelled, and the movement trajectory of the mobile phone is tracked again until the mobile phone is separated from the passenger again, or the passenger takes the mobile phone away from the vehicle.

[0086] In the present embodiment, considering the change of the separation state of the mobile phone, the cancellation of the separation mark is proposed to prevent the mobile phone from being marked with the separation mark when the user retrieves the mobile phone again, and the subsequent interference reminder signal is initiated. At the same time, canceling the separation mark can prevent data redundancy and reduce the work content of the AI ​​processing unit that executes the entire embodiment.

[0087] In a possible embodiment, in response to a vehicle leaving signal from an occupant, sending a reminder signal to the occupant according to a separation mark includes:

[0088] In response to the occupant's leaving signal, a signal display device of the vehicle is awakened according to the separation mark, and the signal display device includes: at least one of a vehicle-mounted display screen, a prompt light and a speaker; and the signal device is controlled to send a reminder signal to the occupant according to a preset prompt strategy.

[0089] In the present embodiment, a variety of feasible ways of sending reminder signals are listed. When the occupants are away from the vehicle, the reminders are mainly given by visual reminders of the reminder lights and auditory reminders of the speakers.

[0090] For example, for visual prompts of warning lights, the door handles of the vehicles include warning lights, which flash frequently when a mobile phone marked as separated is left in the vehicle. Warning lights of different colors and flashing frequencies can be set through the vehicle computer, and the flashing mode of the warning lights can also be customized for the vehicle owner.

[0091] Regarding the auditory prompts of the speaker, while the warning light is flashing, the built-in speaker of the headlight sends a reminder sound such as "ding ding ding" or "Please do not forget your phone in the car". Different reminder tones can be set through the car computer, and the specific content played by the speaker can be customized by the car owner.

[0092] After the passengers return to the car, words or pictures such as "Please do not forget your mobile phone" may be played on the screen of the on-board display device.

[0093] In the present embodiment, a specific description is given of the method for sending reminder signals, mainly in the form of visual reminders and auditory reminders, so that the occupants can notice the abnormality. At the same time, the reminder method can be set actively, that is, the occupants can directly perceive the information that the mobile phone is forgotten based on the unique reminder signal. Even if it cannot be directly perceived by lights or sounds, after the occupants return to the vehicle, they can also perceive that the mobile phone is forgotten through the content displayed on the display device, thereby notifying the occupants in time to forget the mobile phone so that the occupants can retrieve the forgotten mobile phone and avoid the need to return later and waste time.

[0094] In a possible implementation manner, the control signal device sends a reminder signal to the occupant according to a preset reminder strategy, including:

[0095] When the signal display device includes a vehicle-mounted display screen, the vehicle-mounted camera is controlled to identify the location of the mobile phone; and the location of the mobile phone is displayed through the vehicle-mounted display screen.

[0096] By adopting the technical solution of this embodiment, it is possible for passengers to quickly retrieve forgotten mobile phones from the vehicle. When passengers return to the vehicle to look for the forgotten mobile phone due to the prompt signal, it is easy to be unable to determine the location of the mobile phone in a short time, so as to quickly retrieve the forgotten mobile phone. The content of this embodiment is aimed at the tracking function of the trajectory of the mobile phone by the vehicle-mounted camera. While prompting the passengers that the mobile phone is forgotten through the vehicle-mounted display screen, the specific location of the forgotten mobile phone in the car is also identified, such as: the storage space of the left and right doors, near the armrest box, the center console, the storage space on the back of the front seats facing the second row, the first and second rows of seats, and the specific location of the forgotten mobile phone is displayed in the form of text, real-time images or vehicle simulation diagrams, so that users can quickly find the forgotten mobile phone according to the content displayed on the vehicle-mounted display device, reducing the user's search time and saving the user's time.

[0097] In a possible implementation manner, after sending a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal, the method further includes:

[0098] In response to a cancellation signal of the separation mark or a confirmation signal of the occupant that the occupant has not forgotten, the reminder signal is turned off.

[0099] In order to avoid the reminder signal being continuously turned on, the content of this embodiment specifically discloses a method for automatically turning off the reminder signal, that is, when the vehicle-mounted camera confirms that the occupant has returned to the vehicle to retrieve the forgotten mobile phone, the mobile phone comes into contact with the occupant again, the separation mark of the mobile phone is cancelled, and the reminder signal is turned off.

[0100] At the same time, in another usage scenario, if the passenger gets off the vehicle briefly or deliberately leaves the mobile phone in the car, the passenger can send a preset confirmation signal to the vehicle through the sensor to confirm that the mobile phone is left in the car. For example, by pressing the car lock button three times in a row or touching the door handle three times, the passenger can confirm that the mobile phone is left in the car, and the reminder signal will be turned off.

[0101] In the content of this embodiment, the reminder signal is prevented from being continuously outputted. This embodiment discloses a strategy for shutting down the reminder signal. The shutting down strategy is classified, and different methods are used to shut down the reminder signal when the reminder signal is valid, when the occupant returns to the car to retrieve the mobile phone, and when the reminder is invalid and the user confirms to leave the mobile phone in the car, respectively, to ensure the complete operation of the overall strategy of forgetting the mobile phone reminder.

[0102] Based on the method flow chart disclosed in the above embodiment, this embodiment discloses a structure diagram of a forgotten mobile phone reminder device, such as Figure 2 As shown, it includes: an identification module 21, a tracking module 22, a marking module 23 and a reminder module 24;

[0103] The recognition module 21 is used to recognize the handheld items of the occupants by using the vehicle-mounted camera;

[0104] A tracking module 22, for tracking the trajectory of a mobile phone when the handheld object is a mobile phone;

[0105] A marking module 23, used for adding a separation mark to the mobile phone when the mobile phone is separated from the passenger;

[0106] The reminder module 24 is used to send a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal.

[0107] In a possible embodiment, the identification module 21 is specifically configured to:

[0108] Using a vehicle-mounted camera to capture an image including the occupant, and identifying the content of the image;

[0109] Scanning the shape and size of the handheld object, and using the shape and size of the handheld object as first weight information;

[0110] identifying the occupant's control action on the handheld object, and using the control action information as second weight information;

[0111] The first weight information and the second weight information are combined to obtain a recognition result of the handheld object.

[0112] In a possible embodiment, the identification module 21 is specifically configured to:

[0113] Identifying the relative direction of the occupant's face and the handheld object to obtain direction information;

[0114] identifying the occupant's operation action on the handheld object to obtain action information;

[0115] Calculating the duration of the operation action to obtain time information;

[0116] The control action information is generated by combining the direction information, the action information and the time information, and the control action information is used as the second weight information.

[0117] In a possible embodiment, the marking module 23 is further used for:

[0118] When the occupant comes into contact with the mobile phone again, canceling the separation state mark of the mobile phone;

[0119] The trajectory of the mobile phone is tracked again.

[0120] In a possible embodiment, the reminder module 24 is specifically configured to:

[0121] In response to the occupant's leaving signal, waking up the vehicle's signal display device according to the separation mark, the signal display device comprising: at least one of a vehicle-mounted display screen, a warning light, and a speaker;

[0122] The signal device is controlled to send a reminder signal to the occupant according to a preset reminder strategy.

[0123] In a possible embodiment, the reminder module 24 is specifically configured to:

[0124] When the signal display device includes the vehicle-mounted display screen, controlling the vehicle-mounted camera to identify the location of the mobile phone;

[0125] The location of the mobile phone is displayed via the vehicle-mounted display screen.

[0126] In a possible embodiment, the reminder module 24 is further used for:

[0127] In response to a cancellation signal of the separation status mark or a confirmation signal of the occupant that the occupant has not forgotten, the reminder signal is turned off.

[0128] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0129] Based on the above Figure 1 The method shown, and Figure 2 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device that can be configured on the vehicle (such as a new energy vehicle) side, and the device includes at least one processor and a memory that is communicatively connected to the at least one processor; the memory is used to store instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, and the processor is used to execute a computer program to implement the above-mentioned Figure 1 The method shown.

[0130] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0131] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0132] Based on the above Figure 1 The method shown, the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method corresponding to any embodiment is implemented. The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned entity device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the information processing entity device.

[0133] Based on the above electronic device, the embodiment of the present application further provides a vehicle, which may specifically include: Figure 2 The device shown in the figure or the electronic device as described above. The vehicle can be a new energy vehicle or a traditional vehicle.

[0134] In view of the above-mentioned method for reminding a forgotten mobile phone, this embodiment further discloses a structure diagram of a forgotten mobile phone reminder operation system, taking a vehicle with two rows of seats as an example, Figure 3 As shown, the operating system includes: an AI processing unit, an in-cabin camera, a central control display, and a door handle reminder light. For shooting the occupant's mobile phone and accurately reminding the occupant, the in-cabin camera includes: a row of in-cabin cameras and a second row of in-cabin cameras according to the seat arrangement of the vehicle, and the door handle reminder lights include: door handle reminder light-left one, door handle reminder light-left second, door handle reminder light-right one and door handle reminder light-right second according to the number of doors. Among them, the AI ​​processing unit is used to execute the content of the above method embodiment. The first row of in-cabin cameras and the second row of in-cabin cameras are used to capture images of the corresponding seats in the cabin. The central control display is used as a vehicle-mounted display device to display reminder signals and the location of the forgotten mobile phone; the speaker is used to send a preset sound signal as a reminder signal; the four door handle reminder lights are used to send visual light signals to the occupants getting off the corresponding doors.

[0135] Through the description of the above implementation methods, the technicians in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the scheme of this embodiment, compared with the current existing technology, this embodiment first uses the vehicle-mounted camera to identify the handheld items of the occupants; then when the handheld item is a mobile phone, the trajectory of the mobile phone is tracked; then when the mobile phone is separated from the occupants, the mobile phone is marked as a separated state; finally, in response to the occupants' leaving the car signal, a reminder signal is sent to the occupants according to the mobile phone in the separated state. Using the technical scheme of this embodiment, during the occupants' riding process, the vehicle-mounted camera is used to shoot and identify the handheld items of the occupants, and accurately identify them. When it is determined that the handheld item is a mobile phone, the movement trajectory of the mobile phone is tracked, the state of the mobile phone being separated from the occupants is captured, and when the occupants leave the car, a reminder signal is initiated according to the mobile phone in the separated state. Through the technical scheme of this embodiment, the visual recognition technology is used to lock the occupants' mobile phones, monitor the trajectory of the mobile phones, and mark the mobile phones when they are not in contact with the occupants, so that when the occupants leave the car, it is confirmed that the mobile phone is forgotten in the car by the occupants, and a reminder signal is sent to the occupants. By adopting the technical solution implemented in this embodiment, visual recognition technology can be used to confirm the separation relationship between the mobile phone and the occupant, and a reminder signal can be sent when the occupant leaves the car. This can prevent the occupant from having to return to the car to retrieve the mobile phone due to forgetting it in the car, thereby reducing the occupant's time waste.

[0136] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0137] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A method for reminding a forgotten mobile phone, characterized in that: include: Use the vehicle's camera to identify the passenger's handheld items; When the handheld object is a mobile phone, tracking the trajectory of the mobile phone; When the mobile phone is separated from the occupant, adding a separation mark to the mobile phone; In response to a vehicle leaving signal from the occupant, a reminder signal is sent to the occupant according to the separation mark.

2. The method according to claim 1, characterized in that The method of using the vehicle-mounted camera to identify the handheld items of the occupant includes: Using a vehicle-mounted camera to capture an image including the occupant, and identifying the content of the image; Scanning the shape and size of the handheld object, and using the shape and size of the handheld object as first weight information; identifying the occupant's control action on the handheld object, and using the control action information as second weight information; The first weight information and the second weight information are combined to obtain a recognition result of the handheld object.

3. The method according to claim 2, characterized in that The identifying the occupant's control action on the handheld object to obtain second weight information includes: Identifying the relative direction of the occupant's face and the handheld object to obtain direction information; identifying the occupant's operation action on the handheld object to obtain action information; Calculating the duration of the operation action to obtain time information; The control action information is generated by combining the direction information, the action information and the time information, and the control action information is used as the second weight information.

4. The method according to claim 1, characterized in that When the mobile phone is separated from the occupant, after adding a separation mark to the mobile phone, the method further includes: When the occupant comes into contact with the mobile phone again, canceling the separation mark of the mobile phone; The trajectory of the mobile phone is tracked again.

5. The method according to claim 1, characterized in that The step of sending a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal comprises: In response to the occupant's leaving signal, waking up the vehicle's signal display device according to the separation mark, the signal display device comprising: at least one of a vehicle-mounted display screen, a warning light, and a speaker; The signal device is controlled to send a reminder signal to the occupant according to a preset reminder strategy.

6. The method according to claim 5, characterized in that The controlling the signal device to send a reminder signal to the occupant according to a preset reminder strategy includes: When the signal display device includes the vehicle-mounted display screen, controlling the vehicle-mounted camera to identify the location of the mobile phone; The location of the mobile phone is displayed via the vehicle-mounted display screen.

7. The method according to claim 1, characterized in that After sending a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal, the method further includes: In response to a cancellation signal of a separation mark or a confirmation signal of the occupant that the occupant has not forgotten, the reminder signal is turned off.

8. A forgotten mobile phone reminder device, characterized in that: include: An identification module is used to identify objects held by passengers using an on-board camera; A tracking module, used for tracking the trajectory of the mobile phone when the handheld object is a mobile phone; a marking module, used for adding a separation mark to the mobile phone when the mobile phone is separated from the occupant; The reminder module is used for sending a reminder signal to the occupant according to the separation mark in response to the occupant's vehicle leaving signal.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A vehicle, characterized in that: The vehicle is equipped with the device according to claim 8 or the electronic device according to claim 9.

Images